#6549 [Engine] gfx1030 rocBLAS table up to T=16384, and the ssm_alpha/beta shape (ldy 96) it never matched: 98K/128K prompts -4%
open · @Vop3p · 0 comments · View on GitHub
BenchmarksMulti-GPUAMD / HIPModels & quantsDocumentation
Description
## Problem and result Two gaps in the gfx1030 rocBLAS table (`terminal/tools/hip/gfx1030-rocblas-5.6.0.8d1ae90e.txt`) and in the tool that makes it: 1. **ssm_alpha / ssm_beta never matched a row.** `tune_rocblas` assumes ldy = N for every dense shape, but `bf16_proj` writes ssm_alpha and ssm_beta interleaved into one buffer (`prefill.cpp`, `bf16_proj(..., 2 * HV, ...)`), so the engine calls N=48 K=2560 with **ldy=96**. A row matches N, K and ldy exactly, so the shipped `48 2560 48` rows were never used and this shape ran rocBLAS's own pick at every T. The best solution is 1.2x to 6.2x faster than that pick, depending on T (512 tokens: 0.202 -> 0.032 ms). 2. **The buckets stopped at 8192**, but `--prefill 16384` reads prompt chunks of up to 16384 tokens. Larger T use the 8192 row, which is `default` for most shapes. At **T=16384**, rocBLAS's own pick for the GDN gate (N=6144 K=2560) is MT128x256x8: 25.2 ms, against 13.9 ms for its best solution. This happens at 16384 only. At 12288 and 20480 the default pick is fine (20480: 16.7 ms). How I found it: a kernel trace of a 98K-token read showed that the dense FP16 GEMMs take ~39% of the prompt's kernel time on 2x RX 6900 XT. Chunks of exactly 16384 tokens read ~4% slower than chunks of 12288 or 13312. Almost all of that difference was the gate GEMM running on MT128x256x8. Result on 2x RX 6900 XT (Qwen3.8-Flash-Next IQ3_S, layer split 27/21 with DP prompt reads, real-text cold reads). Table off vs this table, two interleaved pairs (B N N B), engine-reported prompt time: | prompt | DP chunks | pair 1 | pair 2 | |---|---|---|---| | 32K | 2 x 16384 | -3.5% | -3.6% | | 64K | 6 x 11008 | 0.0% | -0.1% | | 98K | 6 x 16384 | -4.0% | -4.3% | | 128K | 8 x 16384 | -4.3% | -4.4% | | 200K | 14 x 14336 | -1.6% | -1.7% | The two runs of each table agreed within 0.2% (new table) and 0.4% (old table). ## Related discussion none. The open table PRs for other rocBLAS builds (#3343, #5910, #6313) use the same tool defaults, so they have the same two gaps. I cannot measure those builds here. ## Approach - `tune_rocblas.cpp`: the default shapes now list N, K and ldy as the engine calls them. `48 2560 96` replaces the `48 2560 48` default. The default buckets add 10240, 12288, 14336 and 16384. The full default run takes about 3x as long as before. - `gfx1030-rocblas-5.6.0.8d1ae90e.txt`: 64 rows appended, all measured on an RX 6900 XT against `/opt/rocm/lib` rocBLAS 5.6.0.8d1ae90e (ROCm 10.0.0): - every shape the prompt path calls, at 10240 ... 16384; - `48 2560 96` at every bucket; - `default` rows at 16384 where nothing beats the default, so that a tuned row at a smaller bucket does not reach T values it was not measured at. The rows follow the tool's own acceptance rule (relative L2 <= 2e-3, max abs <= 4 ulp, at least 5% faster). The old rows are unchanged. - `docs/AMD_HIP.md`: the buckets, and `STRATA_ROCBLAS_VERBOSE=1` as the way to list every (T, N, K, ldy) the engine calls, so a table can be checked against them. This is data and a tool default only. No engine code changes, and nothing changes for cards or rocBLAS builds without a table. ## Validation Environment: 2x RX 6900 XT (gfx1030, 16 GB, PCIe 4.0 x8 each), Ryzen 5 5600X, 128 GB DDR4-3200, Ubuntu 26.04, kernel 7.0.0-43, ROCm 10.0.0 (`/opt/rocm/lib` rocBLAS 5.6.0.8d1ae90e). **Passed:** - `tune_rocblas` builds on this branch (`cmake ../engine -DSTRATA_ENABLE_HIP=ON -DCMAKE_HIP_ARCHITECTURES=gfx1030 -DSTRATA_GEOMETRY=flash_next`, target `tune_rocblas`), and `--help` shows the new defaults. - Shape census: `STRATA_ROCBLAS_VERBOSE=1` over 16K/32K/64K/98K/128K/200K reads. It lists 14 shapes. The only one whose ldy differs from N is `48 2560 96`. The largest T is 16384. - Tuning: `tune_rocblas --case ...` for the 57 (shape, T) cells not covered before, plus the gate at 16384/20480/24576. Every written solution passed the tool's output check against the default kernel. - The engine loads the table: `prefill gemm: rocBLAS tuning enabled (176 rows, 111 with a solution, gfx1030, rocBLAS 5.6.0.8d1ae90e)`. - The A/B table above (B N N B, each arm a cold start, 0 stalls). - Standard protocol with this table, in the production concurrent config with the vision encoder on: 21 of 21 requests, 12 of 12 sanity checks (instruction / retrieval / tool calls 4/4), 0 stalls. The 4K and 16K reads were unchanged (1,292 / 2,192 tok/s against 1,287 / 2,194). Their chunks are not 16384. **Failed:** - none observed **Not run:** - The A/B and the protocol ran on my tree: v0.2.17 plus local patches, the same table loader as main. I did not repeat them on current main, and I did not do 10 pairs. The table is data the engine reads unchanged, but the end-to-end numbers above are from my tree. - Other gfx1030 cards, and other rocBLAS builds (5.2.0 / 5.4.0 / 5.8.0 tables). Their owners would need to re-run `tune_rocblas` with the new defaults. - `hip_prefill_rocblas_gemm` (the smoke test). The engine loading the full table is the check I ran. ## Limits - Solution indices are per rocBLAS build, as the docs already say. These rows are for 5.6.0.8d1ae90e only. - The 16384 pathology is a property of rocBLAS's own heuristic on this build. A future rocBLAS may pick well there, and the table would then simply hold a row that is no longer needed. --- Developed with an AI coding assistant; every number above was measured on the machine described in Validation. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
Related on this site
Quick links to related pages.