#5361 [Engine] gfx906 (MI50/MI60/Radeon VII) performance branch — fastest known 4xMI50 Flash-Next numbers
open · @rpcribari · 0 comentários · No GitHub
BenchmarksMulti-GPUAMD / HIPNVIDIA / CUDAModels & quantsDocumentationWindowsLinux
Descrição
## Problem and result gfx906 (MI50 32 GB / MI60 / Radeon VII) is the cheapest way into a big-context MoE rig, but on Qwen3.8-Flash-Next the whole gfx906 ecosystem was stuck around 62 t/s decode and ~1250 t/s prefill-8K (best published llama.cpp fork), and Strata layer-split prefill ran one card at a time on this hardware. This branch (21 commits on 0.2.7) takes a 4x MI50 box to: | metric | before | after | |---|---|---| | decode (single stream, MTP) | ~95 t/s | **125 t/s** | | prefill 8K | 725 t/s | **1250 t/s** | | prefill 32K | 843 t/s | **1766 t/s** | ## Related discussion none — opening as a concrete proposal; happy to rebase onto current main or split into reviewable chunks (kernels / pipeline fix / docs), whichever is easier. ## Approach - **Repacked-GEMM prefill stack** for gfx906: IQ4_NL/IQ3 expert and dense products through a repacked int8 GEMM (vendored from mx-llama.cpp, MIT), FP16-in/out prompt GEMMs, dense MMQ for projections. Every path bitwise-checked against the stock kernels. - **Decode kernels**: MMVQ with activations in LDS + four lanes per cell, IQ GEMV / grouped-expert address-cycle cuts, DPP/`ds_swizzle` reductions replacing `ds_bpermute`, sampler load pipelining, hyper-connection read staging. - **HIP pipeline fix**: reserve the MTP prefill under an in-flight verify window (fixes a decode stall on layer splits). - **`--prefill-pipe` for gfx906**: with the expert cache fully VRAM-resident the per-chunk fixed cost is small, so r ~ 300-450 wins big; without it a layer-split prefill serializes one card at a time. Guidance in `docs/gfx906.md`. All engine-visible behavior is switch-gated (`STRATA_*` env / existing flags); default-on only on the gfx906 build (`-DSTRATA_HIP_GFX906=ON`). No changes outside the HIP/gfx906 paths. ## Validation **Passed:** - `strata-bench`-style throughput runs, revision `int6` (`14f591d4`), 4x MI50 32 GB Gen3 x8, 250 GB RAM, ROCm 7.2.4 container, UD-IQ4_XS pack: decode 125 t/s, pp8K 1250 t/s, pp32K 1766 t/s. Nonce-bearing prompts (no cache hits), warm pass reported, run twice after idle. - Deterministic pass/fail quality harness (8 tasks, temp 0, seed 42): 5/8 — identical to the pre-change baseline, no regressions. - Kernel bitwise checks: repacked GEMM / FP16 GEMM / MMQ paths vs stock kernels via the `*_bench` tools — exact match. **Failed:** - none observed on gfx906. **Not run:** - CUDA / SYCL / Metal builds — no hardware; changes are confined to `ggml/src/ggml-hyperturing` HIP paths and gfx906-gated code, but non-gfx906 HIP is unverified here. - Windows/HIP — Linux only. ## Limits - Branch is based on 0.2.7; main has moved ~4k commits. Rebase offered above. - `--prefill-pipe` optimum is rig-dependent (r 300-450 here); a bad ratio costs decode. - ROCm >= 7 in a container is the reliable runtime on older host kernels (host ROCm 6.3 cannot launch gfx906 kernels); gfx906 Tensile library required.
No site
Links install, modelos, releases.