#5476 [Engine] gfx103x decode: opt-in one-warp MMVQ for K <= 4096 and a per-format expert layout table (STRATA_MMVQ_NW1, STRATA_EXP_TABLE)
open · @Vop3p · 0 评论 · 去 GitHub 看
BenchmarksSetup & installServer & APIMulti-GPUAMD / HIPNVIDIA / CUDAModels & quantsDocumentationWindows
说明
## Problem and result
Two decode-window kernel choices on gfx1030 (RX 6900 XT). Both pick code that is already in the tree; neither adds a kernel.
1. **`STRATA_MMVQ_NW1=1`**: `launch_multi_n` launches the dense multi-column MMVQ (qkv / gate / q / shared expert at T = 2..4) with 4 warps and 2 rows per block. On gfx1030 with cold weights, the existing template `native_mmvq_multi_kernel<F, NCOLS, 1, 1>` (one warp, one row per block) is faster for K <= 4096. The opt-in only changes the dispatch, and K = 6144 (`out`) keeps the default. Kernel times, same binary off/on:
| matrix | T = 2 | T = 3 | T = 4 |
|---|---|---|---|
| qkv Q6_K | 54.2 -> 49.3 us | 57.1 -> 51.4 | 63.1 -> 53.9 |
| qkv Q5_K | 50.0 -> 44.6 | 75.4 -> 45.7 | 88.5 -> 51.5 |
| gate Q4_K | 30.5 -> 26.5 | 37.5 -> 28.0 | 49.8 -> 30.8 |
| q Q6_K | 63.0 -> 59.5 | 66.4 -> 59.6 | 74.3 -> 62.8 |
| shared expert | 5.4 -> 4.7 | 6.4 -> 5.3 | 8.3 -> 5.9 |
Not bitwise: one warp per row sums in a different order, with a relative difference < 1e-5.
2. **`STRATA_EXP_TABLE=1`**: `native_expert_grouped` has one `exp_mode()` for every format, but on gfx1030 the best existing layout depends on the format. A sweep of modes 2..8 against mode 0 with `native_expert_bench` (every result bitwise equal) found gate/up IQ2_S and IQ3_S fastest at mode 8 and down Q2_0 fastest at mode 4. IQ3_XXS gate/up and IQ4_NL down are fastest at mode 0; IQ4_NL down is 7-15% slower at mode 8. No single global mode works, so the table picks per format and leaves every other format at the card's default. A strided call (`grid_groups < cap_groups`) and an explicit `STRATA_EXP_MODE` keep their layout. **Bitwise identical** to mode 0.
The verifier calls `native_expert_grouped` with cap = n x top-k (n = window tokens), and the real group count is only on the device. So the table was measured at that cap (`BENCH_CAP`) over the possible group shapes, for a whole layer's expert step, against mode 0:
| groups in the window | IQ2_S gate/up + Q2_0 down layer | IQ3_S gate/up + IQ4_NL down layer |
|---|---|---|
| every group 1 token (no overlap) | -27 .. -30% | -8 .. -10% |
| groups of 2 | -8 .. -11% | -2 .. -5% |
| 3 tokens share every expert (G 10, T 3) | 0% | +0.4% |
| 4 tokens share every expert (G 10, T 4) | +6.3% | +5.5% |
Full overlap is rare in real decoding. In 266K recorded verify-window routings from our production runs, 59% of windows have 3 tokens, with a median of 23 distinct expert groups out of a possible 30. Of the entries in those windows, 57% sit in a group of 1, 30% in a group of 2 and 13% in a group of 3. 25% of windows have 1 token, 15% have 2, and 1.3% have 4 or 5.
End to end, 2x RX 6900 XT, 10 cold-start interleaved pairs, both variables off vs on, same binary:
| | off (median) | on (median) | paired difference (median) | pairs faster | range |
|---|---|---|---|---|---|
| decode 32K, ms per window | 27.09 | 26.16 | **-3.4%** | 10/10 | -3.8 .. -3.0% |
| decode 4K, ms per window | 28.35 | 28.17 | -0.6% | 10/10 | -0.8 .. -0.4% |
| decode 32K, tok/s | 75.0 | 77.0 | +2.7% | 10/10 | +2.3 .. +3.2% |
| prompt 32K, tok/s | 1,771 | 1,778 | +0.4% | | 0.0 .. +1.0% |
The output text differs from off (MMVQ_NW1 is not bitwise), but each arm produced the same text in all 10 runs, and the number of windows was within one (122/122 at 4K, 126/127 at 32K), so tok/s here is a fair measure. (On the original tree. On the rebased tree this holds at 4K but not at 32K; see the update at the end.)
## Related discussion
none (opt-in kernel selections between existing kernels and layouts, measured on one architecture)
## Approach
- Opt-in rather than a gfx103x default, because both were measured on one chip (two boards of the RX 6900 XT). For the table, the full-overlap case is up to 6% slower. That is rare in decoding, but it is a real regression for that shape.
- `MMVQ_NW1`: a static env check in `launch_multi_n` after the existing `b6_launch` hook, and only for `n_in <= 4096`.
- `EXP_TABLE`: a static env check that is ignored when `STRATA_EXP_MODE` is set. gate/up mode 8 for `gu_type` IQ2_S (22) / IQ3_S (21), down mode 4 for `d_type` Q2_0 (42), otherwise the existing choice. It is skipped for strided calls, like the existing CUDA-layout rule above it, and not compiled for gfx906.
## Validation
Environment: 2x RX 6900 XT (gfx1030, 16 GB, PCIe 4.0 x8 each), Ryzen 5 5600X, 128 GB DDR4-3200, Ubuntu 26.04, kernel 7.0.0-42, ROCm 10.0 runtime (`/opt/rocm/lib`).
**Passed:**
- **Update, on the current branch (main dbb2d168c merged in, plus two small commits: `STRATA_MMVQ_NW1` is now read in HIP builds only, and the settings-register entries for both variables):**
- Build: HIP (flags as below) and CUDA (CUDA 13.2, `-DSTRATA_ENABLE_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=86`, target `strata`) both build clean. The CUDA build was compiled only, not run (no NVIDIA card here); with the HIP guard, no code in this PR changes what a CUDA build does.
- `tools/settings_register.py --check`: clean.
- Quality (KL), both variables on vs off, against a double-precision reference. Setup: this branch, IQ3_S, two-card layer split, 8 teacher-forced chats (a long first message of ~6K to ~103K tokens, then a short scored message; 4,067 scored positions, top-256 logprobs). The reference R is a local build of the same model with double-precision kernels (not part of any PR).
| | KL(R ‖ arm), position-weighted | argmax same as R | top-10 overlap |
|---|---|---|---|
| off | 0.0250 | 95.8% | 92.0% |
| on | 0.0284 | 95.1% | 91.8% |
By chat, "on" is closer to R in 4 of 8 and further in 4 of 8. Paired by chat, the difference is +0.0038 (on - off) with a chat-to-chat spread (sd) of about 0.015, so it is not significant. Two runs of the same arm give bitwise the same logprobs (KL 0), so run-to-run noise cannot serve as the yardstick here. The two arms are KL 0.033 apart, about as far as each is from R: a change in summation order, carried through a long context's KV, moves the logprobs by this much either way. `EXP_TABLE` is bitwise, so all of this comes from `MMVQ_NW1`, which keeps fp32 and changes only the order of the sum (< 1e-5 relative). My reading: no measurable quality change, within the limits of an 8-chat test.
- Build: main 54828c7e + this change, `-DSTRATA_ENABLE_HIP=ON -DCMAKE_HIP_ARCHITECTURES=gfx1030 -DSTRATA_NATIVE_EXPERTS=ON -DSTRATA_PREFILL_MMQ=ON -DSTRATA_Q6K_EXPERTS=ON -DGGML_HIP_GRAPHS=ON -DGGML_HIP_NO_VMM=ON -DSTRATA_BUILD_TESTS=ON`.
- `native_expert_bench` at the verifier's cap: 33 shape/layer/mode combinations, all bitwise equal to mode 0 (layers 1 and 17, one card, 50 iterations).
- The MMVQ microbenchmark (the table in item 1), off/on, built from our production tree with the same change (a small harness around `launch_multi_n`, not part of this PR).
- `ctest -j 1`: 221/224. The three failures are the same on main 54828c7e without this change: `ple_parity` (Q2_0 gguf relative path), `conversation_copy_streams_test` (148 MiB first copy stream) and `isa_guard_check` (CPU iq512 function).
- End to end: the table above. IQ3_S pack, two-card layer split at 27, `--spec 3 --spec-min-p 0.6 --pipeline-windows 1 --kv int8`, parallel 1, our usual HIP env. `benchmark.py` 4K + 32K, 256 tokens each, arms alternating. 0 stalls. (These are the numbers from the original tree; for the rebased tree, see the update at the end.)
**Failed:**
- none observed
**Not run:**
- SYCL / Metal builds. CUDA now builds, and `STRATA_MMVQ_NW1` is limited to HIP (`#if defined(__HIPCC__)`; a CUDA build keeps its default whatever the variable says), which answers the question I had here earlier. `STRATA_EXP_TABLE` is unchanged: the expert layouts are shared code, so a CUDA build reads it too if it is set (gfx906 does not compile it). It was measured on gfx1030 only and is untested on NVIDIA; without the variable nothing changes. I can limit it to HIP as well if you prefer.
- Other gfx103x chips and other RDNA generations. The table is format-specific and measured on gfx1030 only.
- ~~A quality run~~: done, see the KL Update above.
- `ctest` and the 10 pairs on the rebased branch: those lines are from the first version of this PR (main 54828c7e); the rebase only adds the HIP guard and the register entries.
## Limits
- Most of the gain shows up at 32K, where the window carries more T >= 2 dense GEMVs and expert groups. At 4K it is -0.6% per window.
- The table's choice is by format only. The verifier passes cap and not the real group count, so a per-shape choice would need device-side dispatch. I did not do that because full overlap is rare.
## Update 2026-11-20: rebased onto v0.2.16 (main 58ee32277)
Rebased onto current `main` (`v0.2.16-4-g58ee32277`). The change itself is unchanged: the files moved to `engine/`, and the only conflict was the generated `docs/SETTINGS.md`, which I regenerated with `tools/settings_register.py --docs` (`--check` clean). Three commits, no merge commit.
Re-measured on the rebased tree: same box, now kernel 7.0.0-43. 10 cold-start interleaved pairs (off, on, on, off), both variables off vs on, same binary. IQ3_S, layer split 27/21, `--spec 3 --spec-min-p 0.6 --pipeline-windows 1 --kv int8`, parallel 1, temperature 0, `STRATA_HC_PERSIST=1`, `benchmark.py` 4K + 32K, 256 tokens. 0 stalls.
| | drafts per window (off / on) | ms per window | ms per verified token | decode tok/s | prompt tok/s |
|---|---|---|---|---|---|
| 4K | 1.43 / 1.46 | 28.96 -> 28.43, **-1.8%** (10/10, -2.8 .. -1.6%) | **-3.2%** (10/10) | +4.3% (10/10) | +0.4% |
| 32K | 1.41 / 1.28 | 27.72 -> 25.75, -7.1% (10/10, -7.2 .. -6.4%) | -1.7% (10/10) | -3.0% (0/10) | +0.9% |
A correction to the paragraph above that calls tok/s "a fair measure": on this tree that holds at 4K but not at 32K. The 32K text with both variables on is a different text, and the drafts on it are accepted less often. There are 132 windows instead of 119, and fewer drafts are offered per window. So 32K tok/s is lower (-3.0%) even though every window is cheaper, and ms per window overstates the gain, because a window with fewer drafts costs less. "ms per verified token" (decode ms / (windows + drafts offered)) corrects in the other direction. Window cost is mostly fixed, so that column over-corrects. I read 32K as somewhere between -1.7% and -7.1% per unit of work. 4K is the clean case: drafts per window are about equal there, and both measures agree (-1.8% / -3.2%). Each arm still gave the same text in all 10 runs.
- `ctest -j 1`: 289/291. The two failures are this machine's existing ones, as above: `ple_parity` (Q2_0 GGUF not on this box) and `conversation_copy_streams_test` (ROCm 10's 148 MiB first copy stream).
- Standard protocol with both variables on: 21/21, 12 of 12 sanity checks (instruction / retrieval / tool calls 4/4), 0 stalls. The status line says "nonconforming" only because the bench could not read a commit from a git worktree ("the Strata commit is unknown"). No measurement was affected.
- Build flags (v0.2.16 has no `STRATA_PREFILL_MMQ`): `-DSTRATA_ENABLE_HIP=ON -DCMAKE_HIP_ARCHITECTURES=gfx1030 -DSTRATA_BUILD_TESTS=ON -DSTRATA_AMD_COMMUNITY_TUNING=ON -DSTRATA_GEOMETRY=flash_next -DSTRATA_NATIVE_EXPERTS=ON -DSTRATA_OWN_CPU_QUANT=ON -DSTRATA_Q6K_EXPERTS=ON`, ROCm 10.0 clang.
- Rebased once more on 2026-11-21 onto `main` 3142b66b9 (`v0.2.16-281`, 0.2.17 plus 3 commits), because the generated `docs/SETTINGS.md` conflicted again. No code conflict. On that tip: HIP build clean, `tools/settings_register.py --check` clean, `ctest -j 1` 311/313 (the same two failures as above). The speed and gate numbers in this update were measured on 58ee32277 and were not re-run on the new tip.
---
Developed with an AI coding assistant; every number above was measured on the machine described in Validation.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
本站相关内容
相关页面的快捷入口。