#6532 [Engine] prefill: opt-in STRATA_I8_DEFER - unpicked streamed entries no longer cut the int8 expert groups into fragments (gfx1030 4K prompt +3.6%)

open · @Vop3p · 0 comments · View on GitHub

BenchmarksSetup & installMulti-GPUAMD / HIPNVIDIA / CUDAModels & quantsDocumentationWindows

Description

## Problem and result

On gfx1030, the prompt read's int8 expert kernels run on a group of 16 experts at a time (`i8_group.hpp`). In a streamed walk, a group was usually cut into much smaller launches. The streamed walk hands an entry's ring slot back as soon as it reaches an entry that no token of the chunk routes to (`release_to` in `engine/src/prefill/prefill.cpp`). Before that it flushes the open group, so that the group's own slots get their events first. For the MMQ walk this flush only gathers: the GEMM still waits for the full group. For the int8 walk the flush **is** the launch. So every unpicked streamed entry ended the open group early, and a group of 16 experts became several launches of one to a few experts each, with tens of rows. That is the int8 kernels' least efficient shape: they fall to tile 2/3 with mostly empty tiles.

Kernel trace of one 4K-token cold read on 2x RX 6900 XT (IQ3_S pack, layer split 27/21), IQ2_S gate/up only (20 layers):

| | launches | launches per layer | tile 2 launches | int8 expert GEMM, all formats, both cards |
|---|---|---|---|---|
| before | 1,400 | 70 | 401 (1-2 experts, tens of rows each) | 1,073 ms |
| `STRATA_I8_DEFER=1` | 567 | 28.4 | 32 | 972 ms (-9.4%) |

With 16-expert groups there would be 32 launches per layer, so 28.4 is the group size the walk was meant to have.

GPU idle time inside the prompt window was the same before and after (and the same as on the MMQ path of a v0.2.15 build). The fragments did not leave gaps between launches. They made the expert kernels themselves slower. This may be what #6264 sees ("MoE GEMM gate/up 3.4x slower" on an RX 6800 since the int8 kernels became the default). I have not checked it on that card.

End to end on current `main` (6e38f588a + this change), 10 interleaved cold-start pairs, off vs on, same binary:

Real-text cold reads (one per size, in order 2K, 4K, 16K), engine-reported prompt tok/s, the production environment of this machine on stock main (switches main does not know are ignored), layer split 27/21, IQ3_S pack:

| prompt | off (median) | on (median) | paired difference (median) | pairs faster | range |
|---|---|---|---|---|---|
| 2K | 741 | 740 | +0.1% | 6/10 | -1.2 .. +0.7% |
| 4K | 1,121 | 1,168.5 | **+3.6%** | **10/10** | +2.8 .. +4.5% |
| 16K | 1,686 | 1,714.5 | **+1.9%** | **10/10** | +1.4 .. +2.6% |

A 2K chunk has few unpicked streamed entries to begin with, so it has little to gain.

On our own production-derived tree (v0.2.17 + local patches), the same switch gave +3.8% / +3.8% at 4K and +1.3% / +1.5% at 16K (two pairs each). With it, the int8 path reads prompts at the speed of the v0.2.15 MMQ path on this machine: 4K -0.7%, 16K +0.1%, and 2K +13%.

## Related discussion

#6264 (RDNA2 prefill regression, MoE gate/up GEMM) and #6269 (gfx103x after the MMQ removal). This PR is an opt-in fix in the prompt path's group walk. It changes no kernel and no default.

## Approach

- In `release_to`, when the layer runs on the int8 groups (`i8_l`) and the open group holds ring slots, the unpicked entry's slot gets its event at once. The flush is skipped, and `give_back` is not called for it. The walk's existing `gg_nslots == 0 -> give_back(consumed)` then hands back everything up to there once the group has launched normally (the flush records the group's slot events, as before). No slot of the open group can be refilled before the group has read it.
- Deadlock guard: while entries are held back the stager cannot run further ahead. If the entries not yet handed back plus a full group would come within two entries of the ring size, the old early launch is used. (`given_upto` records the last count handed back.)
- Opt-in `STRATA_I8_DEFER=1`, read once. The MMQ walk and the resident (`stage_of`) walk are untouched. The settings register and `docs/SETTINGS.md` have the entry.
- Same kernels and same rows. Only the launch boundaries move, and every int8 tile computes a (row, entry) with the same terms in the same order (the existing bit-identity of the tiles). So the output is bit-identical.

## Validation

Environment: 2x RX 6900 XT (gfx1030, 16 GB, PCIe 4.0 x8 each), Ryzen 5 5600X, 128 GB DDR4-3200, Ubuntu 26.04, kernel 7.0.0-43, ROCm 10.0 runtime.

**Passed:**
- Build: HIP (`-DSTRATA_ENABLE_HIP=ON -DCMAKE_HIP_ARCHITECTURES=gfx1030 -DSTRATA_BUILD_TESTS=ON -DSTRATA_AMD_COMMUNITY_TUNING=ON -DSTRATA_GEOMETRY=flash_next -DSTRATA_NATIVE_EXPERTS=ON -DSTRATA_OWN_CPU_QUANT=ON -DSTRATA_Q6K_EXPERTS=ON`, ROCm 10.0 clang). `tools/settings_register.py --check`: clean.
- `ctest -j 1`: 311 of 313. The two failures are this machine's existing ones: `ple_parity` (its Q2_0 GGUF is not on this box) and `conversation_copy_streams_test` (ROCm 10 takes 148 MiB for a first copy stream).
- Bit-identity (on the v0.2.17 tree with local patches; deterministic settings: CPU share 0, `--adapt-every 0`, fixed `--pcie-frac`, `--pipeline-windows 1`, temperature 0): 4K and 32K prompts, 256 generated tokens each. The text is the same with and without the switch, and the decode speed is unchanged.
- Kernel trace: the table above.
- The 10 pairs above, on main 6e38f588a + this change.
- Standard protocol with `STRATA_I8_DEFER=1`: 21 of 21 requests, 12 of 12 sanity checks (instruction / retrieval / tool calls 4/4), 0 stalls. The status line says "nonconforming" only because the bench could not read a commit from a git worktree.
- 0 stalls in every run.

**Failed:**
- none observed

**Not run:**
- CUDA / SYCL / Metal builds and runs. The change sits in host code shared by the CUDA and HIP prompt paths and only acts where the int8 groups run (`i8_l`). I have no NVIDIA card here.
- A single-card setup and other gfx103x cards (#6264's RX 6800 in particular).
- Chunks where the deadlock guard actually falls back. It did not happen with ring sizes as configured here; the guard keeps today's behaviour when it does.

## Limits

- The stager is held back by up to one group's entries while a group is open. I saw no stall and no slower copy in the runs above, but a much smaller prompt ring would hit the guard more often.
- Only the int8 walk is changed. The MMQ walk never had the problem.

---
Developed with an AI coding assistant; every number above was measured on the machine described in Validation.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Related on this site

Quick links to related pages.