#5361 [Engine] gfx906 (MI50/MI60/Radeon VII) performance branch — fastest known 4xMI50 Flash-Next numbers

open · @rpcribari · 0 Kommentare · Auf GitHub

BenchmarksMulti-GPUAMD / HIPNVIDIA / CUDAModels & quantsDocumentationWindowsLinux

Beschreibung

## Problem and result

gfx906 (MI50 32 GB / MI60 / Radeon VII) is the cheapest way into a big-context MoE rig, but on Qwen3.8-Flash-Next the whole gfx906 ecosystem was stuck around 62 t/s decode and ~1250 t/s prefill-8K (best published llama.cpp fork), and Strata layer-split prefill ran one card at a time on this hardware.

This branch (21 commits on 0.2.7) takes a 4x MI50 box to:

| metric | before | after |
|---|---|---|
| decode (single stream, MTP) | ~95 t/s | **125 t/s** |
| prefill 8K | 725 t/s | **1250 t/s** |
| prefill 32K | 843 t/s | **1766 t/s** |

## Related discussion

none — opening as a concrete proposal; happy to rebase onto current main or split into reviewable chunks (kernels / pipeline fix / docs), whichever is easier.

## Approach

- **Repacked-GEMM prefill stack** for gfx906: IQ4_NL/IQ3 expert and dense products through a repacked int8 GEMM (vendored from mx-llama.cpp, MIT), FP16-in/out prompt GEMMs, dense MMQ for projections. Every path bitwise-checked against the stock kernels.
- **Decode kernels**: MMVQ with activations in LDS + four lanes per cell, IQ GEMV / grouped-expert address-cycle cuts, DPP/`ds_swizzle` reductions replacing `ds_bpermute`, sampler load pipelining, hyper-connection read staging.
- **HIP pipeline fix**: reserve the MTP prefill under an in-flight verify window (fixes a decode stall on layer splits).
- **`--prefill-pipe` for gfx906**: with the expert cache fully VRAM-resident the per-chunk fixed cost is small, so r ~ 300-450 wins big; without it a layer-split prefill serializes one card at a time. Guidance in `docs/gfx906.md`.

All engine-visible behavior is switch-gated (`STRATA_*` env / existing flags); default-on only on the gfx906 build (`-DSTRATA_HIP_GFX906=ON`). No changes outside the HIP/gfx906 paths.

## Validation

**Passed:**
- `strata-bench`-style throughput runs, revision `int6` (`14f591d4`), 4x MI50 32 GB Gen3 x8, 250 GB RAM, ROCm 7.2.4 container, UD-IQ4_XS pack: decode 125 t/s, pp8K 1250 t/s, pp32K 1766 t/s. Nonce-bearing prompts (no cache hits), warm pass reported, run twice after idle.
- Deterministic pass/fail quality harness (8 tasks, temp 0, seed 42): 5/8 — identical to the pre-change baseline, no regressions.
- Kernel bitwise checks: repacked GEMM / FP16 GEMM / MMQ paths vs stock kernels via the `*_bench` tools — exact match.

**Failed:**
- none observed on gfx906.

**Not run:**
- CUDA / SYCL / Metal builds — no hardware; changes are confined to `ggml/src/ggml-hyperturing` HIP paths and gfx906-gated code, but non-gfx906 HIP is unverified here.
- Windows/HIP — Linux only.

## Limits

- Branch is based on 0.2.7; main has moved ~4k commits. Rebase offered above.
- `--prefill-pipe` optimum is rig-dependent (r 300-450 here); a bad ratio costs decode.
- ROCm >= 7 in a container is the reliable runtime on older host kernels (host ROCm 6.3 cannot launch gfx906 kernels); gfx906 Tensile library required.

Mehr auf der Site

Links zu Install, Modellen, Releases.