#6269 [Engine] gfx103x: prompt throughput gaps left by the MMQ removal (dense projections, IQ4_XS down)
open · @schmittcn · 6 comments · View on GitHub
BenchmarksSetup & installAMD / HIPDocumentationLinux
Description
Since own-stack step 10 (3e26b4e15) the HIP build has no MMQ path. For the official models on gfx1030 the loss is small, because MMQ had been off by default since 0.2.14 and Strata's own int8 expert kernels cover the MoE layers. Two gaps remain, though, and one setting no longer does anything. **Setup:** RX 6900 XT 16 GB (gfx1030), PCIe 4.0 x16, Ryzen 9 5950X, 64 GB RAM, Arch Linux (Omarchy, kernel 7.2.15), ROCm 7.2.4, performance power profile. Each arm ran 3 interleaved passes and the figures are medians. The 40K figure is 3 cold 40,207-token prompts per pass. ### 1. Dense projections: `STRATA_DENSE_MMQ` is now a no-op Qwen3.6-35B-A3B UD-Q4_K_XL, stock 0.2.16 vs a stock v0.2.15 build with MMQ and `STRATA_DENSE_MMQ=1`: | | 0.2.16 | v0.2.15 + MMQ + DENSE_MMQ | |---|---|---| | 4K prompt tok/s | 1342 | 2310 (+72%) | | 16K prompt tok/s | 1063 | 1598 (+50%) | | 40K prompt tok/s (cold) | 724 | 930 (+28%) | | agent turn first token | 4.42 s | 2.70 s | Output stays the same: teacher-forced over 201 steps, mean KL is 0.0029 (p99 0.024) and top-1 agrees 98.5%. The decode differences between the two versions (58.5 vs 43.2 tok/s chat) come from 0.2.16's MTP-on-sampled change, not from MMQ. `Gemm::native_mmq` now returns false because `mmq::built()` is false. `STRATA_DENSE_MMQ` is still in docs/ and the settings register, so setting it silently does nothing. **Ask:** either an int8 (dp4a / sdot4) dense prompt path of Strata's own for gfx103x, or retire the setting from docs and the register so nobody tunes a dead knob. ### 2. IQ4_XS as a down format Strata's int8 expert kernels have no IQ4_XS down format. #6197 adds IQ4_XS to the int8 expert kernels and rebases cleanly onto 0.2.16. With it, a fine-tune whose experts are IQ4_XS (Ornith 1.5 IQ4_XS) reaches 719 tok/s at 40K, against 956 tok/s with the old MMQ build. The rest of that gap is the dense projections (point 1): | Ornith 1.5 | 0.2.16 + #6197 | 0.2.15 + MMQ | |---|---|---| | IQ4_XS 4K / 16K / 40K | 1335 / 1061 / 719 | 2411 / 1650 / 956 | | Q6_K 4K / 16K / 40K | 1226 / 988 / 694 | 1914 / 1406 / 861 | ### 3. Related In #6067, an RX 6800 (gfx1030) report saw 910 → 580 tok/s after 0.2.14, when MMQ went off by default. That is consistent with the dense gap above. I'm glad to test any gfx103x kernel. I can also adapt #3926's measurement setup.
Related on this site
Quick links to related pages.