#6549 [Engine] gfx1030 rocBLAS table up to T=16384, and the ssm_alpha/beta shape (ldy 96) it never matched: 98K/128K prompts -4%

open · @Vop3p · 0 comments · View on GitHub

BenchmarksMulti-GPUAMD / HIPModels & quantsDocumentation

Description

## Problem and result

Two gaps in the gfx1030 rocBLAS table (`terminal/tools/hip/gfx1030-rocblas-5.6.0.8d1ae90e.txt`) and in the tool that makes it:

1. **ssm_alpha / ssm_beta never matched a row.** `tune_rocblas` assumes ldy = N for every dense shape, but `bf16_proj` writes ssm_alpha and ssm_beta interleaved into one buffer (`prefill.cpp`, `bf16_proj(..., 2 * HV, ...)`), so the engine calls N=48 K=2560 with **ldy=96**. A row matches N, K and ldy exactly, so the shipped `48 2560 48` rows were never used and this shape ran rocBLAS's own pick at every T. The best solution is 1.2x to 6.2x faster than that pick, depending on T (512 tokens: 0.202 -> 0.032 ms).
2. **The buckets stopped at 8192**, but `--prefill 16384` reads prompt chunks of up to 16384 tokens. Larger T use the 8192 row, which is `default` for most shapes. At **T=16384**, rocBLAS's own pick for the GDN gate (N=6144 K=2560) is MT128x256x8: 25.2 ms, against 13.9 ms for its best solution. This happens at 16384 only. At 12288 and 20480 the default pick is fine (20480: 16.7 ms).

How I found it: a kernel trace of a 98K-token read showed that the dense FP16 GEMMs take ~39% of the prompt's kernel time on 2x RX 6900 XT. Chunks of exactly 16384 tokens read ~4% slower than chunks of 12288 or 13312. Almost all of that difference was the gate GEMM running on MT128x256x8.

Result on 2x RX 6900 XT (Qwen3.8-Flash-Next IQ3_S, layer split 27/21 with DP prompt reads, real-text cold reads). Table off vs this table, two interleaved pairs (B N N B), engine-reported prompt time:

| prompt | DP chunks | pair 1 | pair 2 |
|---|---|---|---|
| 32K | 2 x 16384 | -3.5% | -3.6% |
| 64K | 6 x 11008 | 0.0% | -0.1% |
| 98K | 6 x 16384 | -4.0% | -4.3% |
| 128K | 8 x 16384 | -4.3% | -4.4% |
| 200K | 14 x 14336 | -1.6% | -1.7% |

The two runs of each table agreed within 0.2% (new table) and 0.4% (old table).

## Related discussion

none. The open table PRs for other rocBLAS builds (#3343, #5910, #6313) use the same tool defaults, so they have the same two gaps. I cannot measure those builds here.

## Approach

- `tune_rocblas.cpp`: the default shapes now list N, K and ldy as the engine calls them. `48 2560 96` replaces the `48 2560 48` default. The default buckets add 10240, 12288, 14336 and 16384. The full default run takes about 3x as long as before.
- `gfx1030-rocblas-5.6.0.8d1ae90e.txt`: 64 rows appended, all measured on an RX 6900 XT against `/opt/rocm/lib` rocBLAS 5.6.0.8d1ae90e (ROCm 10.0.0):
  - every shape the prompt path calls, at 10240 ... 16384;
  - `48 2560 96` at every bucket;
  - `default` rows at 16384 where nothing beats the default, so that a tuned row at a smaller bucket does not reach T values it was not measured at.
  
  The rows follow the tool's own acceptance rule (relative L2 <= 2e-3, max abs <= 4 ulp, at least 5% faster). The old rows are unchanged.
- `docs/AMD_HIP.md`: the buckets, and `STRATA_ROCBLAS_VERBOSE=1` as the way to list every (T, N, K, ldy) the engine calls, so a table can be checked against them.

This is data and a tool default only. No engine code changes, and nothing changes for cards or rocBLAS builds without a table.

## Validation

Environment: 2x RX 6900 XT (gfx1030, 16 GB, PCIe 4.0 x8 each), Ryzen 5 5600X, 128 GB DDR4-3200, Ubuntu 26.04, kernel 7.0.0-43, ROCm 10.0.0 (`/opt/rocm/lib` rocBLAS 5.6.0.8d1ae90e).

**Passed:**
- `tune_rocblas` builds on this branch (`cmake ../engine -DSTRATA_ENABLE_HIP=ON -DCMAKE_HIP_ARCHITECTURES=gfx1030 -DSTRATA_GEOMETRY=flash_next`, target `tune_rocblas`), and `--help` shows the new defaults.
- Shape census: `STRATA_ROCBLAS_VERBOSE=1` over 16K/32K/64K/98K/128K/200K reads. It lists 14 shapes. The only one whose ldy differs from N is `48 2560 96`. The largest T is 16384.
- Tuning: `tune_rocblas --case ...` for the 57 (shape, T) cells not covered before, plus the gate at 16384/20480/24576. Every written solution passed the tool's output check against the default kernel.
- The engine loads the table: `prefill gemm: rocBLAS tuning enabled (176 rows, 111 with a solution, gfx1030, rocBLAS 5.6.0.8d1ae90e)`.
- The A/B table above (B N N B, each arm a cold start, 0 stalls).
- Standard protocol with this table, in the production concurrent config with the vision encoder on: 21 of 21 requests, 12 of 12 sanity checks (instruction / retrieval / tool calls 4/4), 0 stalls. The 4K and 16K reads were unchanged (1,292 / 2,192 tok/s against 1,287 / 2,194). Their chunks are not 16384.

**Failed:**
- none observed

**Not run:**
- The A/B and the protocol ran on my tree: v0.2.17 plus local patches, the same table loader as main. I did not repeat them on current main, and I did not do 10 pairs. The table is data the engine reads unchanged, but the end-to-end numbers above are from my tree.
- Other gfx1030 cards, and other rocBLAS builds (5.2.0 / 5.4.0 / 5.8.0 tables). Their owners would need to re-run `tune_rocblas` with the new defaults.
- `hip_prefill_rocblas_gemm` (the smoke test). The engine loading the full table is the check I ran.

## Limits

- Solution indices are per rocBLAS build, as the docs already say. These rows are for 5.6.0.8d1ae90e only.
- The 16384 pathology is a property of rocBLAS's own heuristic on this build. A future rocBLAS may pick well there, and the table would then simply hold a row that is no longer needed.

---
Developed with an AI coding assistant; every number above was measured on the machine described in Validation.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Related on this site

Quick links to related pages.