#2492 gfx12: empty workgroups past the full-die fill stall in the decode matvecs (grid_pad)

Descrição

## Title
gfx12: empty workgroups past the full-die fill stall in the decode matvecs (`grid_pad`)

Issue: none open on this (searched issues and PRs for the stall, 2048-wave grids and R9700 decode). The cause is
ROCm/legacy-rocm-build#6689.

## Summary
On RDNA4 with the full die (R9700 / 9070 XT: 32 WGPs = 2,048 wave slots), a dispatch whose waves land at, or just
below, a whole multiple of the slots runs ~8–14 µs longer than one a few waves bigger. Reproduced on an R9700 with
ROCm 7.14.1 and 7.2.4 alike: 512-thread blocks with a pointer argument and LDS, 118–124 and 129–133 blocks 2.24 µs
per launch, 125–128 blocks 16.4 µs. Several of Strata's decode matvecs hit it. This PR launches them with a few extra
workgroups that return at once.

Measured on one R9700 (gfx1201), Linux, ROCm 7.14.1, against main 296b59d2 (after v0.1.44.1). One launch in a
hipGraph, weights rotated past the 64 MB Infinity Cache:

| launch | main | this PR |
|---|---|---|
| `native_small_mmvq` 6144→2560 (ssm_out / attn_output), 1 column, 2560 blocks | 41.3 µs | 30.2 µs |
| `native_small_mmvq` 2560→512 (attn_k / attn_v / indexer q), 1 column, 512 blocks | 16.5 µs | 5.0 µs |
| `bf16_f32_mmvf_kernel` 2560→512 (router), 512 blocks of 256 threads | 19.7 µs | 7.2 µs |
| `bf16_f32_mmvf_kernel` 2560→1024 | 24.5 µs | 11.7 µs |

Server: UD-IQ4_XS, `--spec 4` with MTP, kv int8, `--max-context 32768`, the gfx12 decode switches and
`STRATA_ROUTE_RESIDENT=1`. Main, this PR with `STRATA_GRID_PAD=0`, and this PR; 3 starts each, alternating, 3 prompts
× 512 tokens, greedy. Prompts 1 and 2 are shown: prompt 3's text varied between starts, see Extra Notes; one start's
prompt 2 also differed and is left out.

| | verify ms/window (p1 / p2) | GPU wait (p1 / p2) | tok/s (p1 / p2) |
|---|---|---|---|
| main | 21.84–22.18 / 20.97–21.28 | 17.68–17.73 / 17.15–17.23 | 59.0–60.4 / 71.1–72.6 |
| this PR, `STRATA_GRID_PAD=0` | 21.75–22.19 / 20.88–20.93 | 17.67–17.69 / 17.17 | 58.7–59.4 / 69.9–72.6 |
| this PR | 21.28–21.64 / 20.33–20.36 | 17.14–17.18 / 16.55–16.57 | 60.5–61.4 / 73.9–74.8 |

## What changed
- `include/strata/kernels/grid_pad.hpp` (new): `grid_pad(blocks, threads)` returns the grid to launch — `blocks`
  plus a few empty blocks when the launch's waves fall within the last max(16, 4·waves-per-block) slots below a
  multiple of the slots, or within 16·k waves past the k-th multiple (at most 256), fitted to the measurements.
  gfx12 only (`gcnArchName` gfx1200 / gfx1201); other HIP cards and CUDA get the grid unchanged. `STRATA_GRID_PAD=0`
  turns it off, `=1` applies it on any HIP card.
- `src/kernels/cuda/native_mmvq.cu`: `native_small_mmvq_kernel` and `native_mmvq_multi_kernel` (with the LFUSE pair)
  return at once in a block past their real rows; their launchers go through `grid_pad`. The math is untouched.
- `src/kernels/cuda/native_bf16.cu`: `bf16_f32_mmvf_kernel` takes `n_out` and returns in a block past it; its
  launcher goes through `grid_pad`.
- `src/kernels/grid_pad_test.cpp` + `CMakeLists.txt`: `grid_pad_test`, the rule on the measured cases (no GPU).
- `docs/AMD_HIP.md`: an entry under RDNA4's speed switches.

Left alone: every other kernel and launch, the default on every non-gfx12 card, CUDA, and the SYCL port (it builds its
own migrated `.dp.cpp` copies of these kernels, so none of the changed files reach it).

**The default output stays byte-identical.** Every padded launcher on random inputs (`native_mmvq` at 1–8 columns on
the model's dense shapes, the LFUSE pair, `bf16_f32_mmvf`; 12.5 M values in 153 segments), `STRATA_GRID_PAD=0` vs on
in separate processes, with `STRATA_TSUM` 0 and 1: the outputs are byte-identical, and nothing past them is written.

**Builds:** HIP (gfx1201, system ROCm 7.14.1): ctest 112/113 — `ple_parity` fails as on main (needs the Q2_0
fixture) — and the matvec parity tests also with `STRATA_GRID_PAD=0` (52/52). CUDA (nvcc 13.4, sm_120): compiles and
links, `grid_pad_test` passes (no NVIDIA GPU here to run it). SYCL: not built — none of the changed files are in its
build.

## Extra Notes
- The early return also changes `native_mmvq_multi_kernel`'s code: the Q6_K head (2560→248320) at 2–8 columns gets
  3–11% faster (T=4: 1158 → 1037 µs), attn_gate / qkv / attn_q at 4–6 columns 1.0–1.8 µs (2–3%) slower; on the
  server the two cancel (the `STRATA_GRID_PAD=0` row above matches main).
- Served answers, `STRATA_ROUTE_RESIDENT` off, the same request first after start: 4 starts of main and 8 of this PR
  (4 with the padding off, 4 on): 11 of 12 the same text, and the one that differs is a main start. That matches the
  "Known" line in docs/AMD_HIP.md (a HIP run's greedy output sometimes differs from another start's); the
  kernel-level check above is what shows that this PR does not change the output.
- On a 0.1.43.1-based build with `--kv-grow --vram-elastic` (the open HIP VMM PRs) the same server A/B gave
  verify −1.0 ms/window (−4.5%) and +3.4…4.7% tok/s, 27 of 27 answers byte-identical. There the early return alone
  also took away a stall of the multi kernel (attn_gate at 2–4 columns 41–44 → 31 µs) that main avoids through
  `q8_0_rows1`.
- Not tested: running it on CUDA; other RDNA4 dies (the 56-CU RX 9070 did not show the stall in the ROCm report; the
  padding there costs a few empty blocks); Windows.

🤖 Generated with [Claude Code](https://claude.com/claude-code)