#6290 [Engine] gfx1100 + UD-IQ4_XS-v31: ask for a Q5_0-down route in the fused prompt path (our int8 extension measures +55% prefill at 114K)

open · @Jerry-Lee661 · 0 comments · View on GitHub

BenchmarksMulti-GPUAMD / HIPModels & quants

Description

### Area

[Engine]  (speed, kernels, models, the server and API)

### What do you want to do?

Please give gfx1100 (RX 7900 XTX) a Q5_0-down route in the fused prompt path for UD-IQ4_XS-v31 sets (33 of 48 layers have Q5_0 down, with IQ3_XXS or IQ4_XS gate/up). On gfx1100 those layers currently leave the fused path, and the existing int8 Q5_0 down instance (`native_w11_kernel<T_Q5_0,false>`) appears to be unused on gfx11.

Environment: 2x RX 7900 XTX 24 GB (gfx1100), Xeon E5-2686 v4, 121 GiB DDR3, Ubuntu 26.04.1, ROCm runtime 10.4.01 (hipBLASLt table 100401), engine built from source on a 0.1.59-lineage merge (upstream v0.2.13 engine + the rdna3 fork kernels).

What we measured, with a local extension that opens `d_covered_on`'s Q5_0 for gfx1100 behind `STRATA_PF_FUSED_Q50=1` and `STRATA_PF_F16=0` (pure int8):

- `hipFuncGetAttributes(native_w11_kernel<T_Q5_0,false>)`: LDS 24,832 B, 218 registers (the IQ3_XXS gate/up reference is 44,544 B). The instance sets up and runs.
- 32K and 114K prompts complete with no hang and no crash; the int8 route is stable in production (load 100-130 s with arena reuse, decode 90-95 tok/s on a 1024-token done-line).
- Same-machine A/B at 114K (TP2, UD-IQ4_XS-v31), three medians: int8 + hipBLASLt 100401 table 2903-3155 tok/s; F16 + table 3025-3094; baseline MMQ without table 1931.
- Solo single-card 32K prefill, same machine: IQ3_S pack 2918-2950 tok/s vs UD-IQ4_XS Q5_0-int8 2088. The ~30% gap looks like missing launch tuning rather than structure: Q5_0 has no `occ_d` entry and no wave-cap / ILP characterization, unlike the IQ3_S path.

The request, concretely:

1. Extend `q50_on()` to gfx1100 (today it returns false for `!w12 && !g1151`), either with the same F16-gated rule as gfx1151, or with the int8 rule like gfx12. Our measurements suggest the int8 instance is correct on gfx11; with `STRATA_PF_F16=2` (gfx1100's default on the 0.1.59 line) the F16 route also covers these layers now that `f16_d()` includes Q5_0, so a F16-gated rule may be the smaller change.
2. Add an `occ_d` entry (and any needed wave-cap / ILP tuning) for Q5_0 on gfx11, or publish the recommended launch parameters for the instance.

Related history: #5296 (gfx1151, UD-IQ4_XS Q5_0-down leaving the fused prompt path; addressed there via the F16 route in round 0217). We hit the old `d_covered` guard's `std::exit(1)` on our 0.1.59 tree with `STRATA_PF_F16=2`; main's `f16_d()` now includes Q5_0, so I believe that part is already fixed upstream and only record it as context, not as a request.

We can run builds and A/Bs on this machine if a branch needs gfx1100 numbers (solo or TP2; both forms are tested here).

### What do you do today instead?

On gfx1100 we run our local extension (`STRATA_PF_FUSED_Q50=1` + `STRATA_PF_F16=0`) on the 0.1.59-lineage tree: it is stable and gives the numbers above. Without the extension, the Q5_0-down layers take the gather + FP16 dequantize + hipBLASLt route (the baseline row above, ~30% slower for the same pack). On the 0.1.59 line with `STRATA_PF_F16=2` and no extension, the first long prompt exits the engine through the old guard.

Related on this site

Quick links to related pages.