#6429 [Benchmark] hipBLASLt tuning table by default on gfx1100 (RX 7900 XTX, IQ3_S): speed, decode and logits
closed · @superbaenawa · 5 comentarios · En GitHub
BenchmarksSetup & installAMD / HIPModels & quantsDocumentationWindowsLinux
Descripción
### What did you measure, or what went wrong #5666 reported the speed on 0.2.13 and was closed as a benchmark record, without a decision on the default. This asks for that decision, with the decode and answer measurements it depends on. **Summary.** On an RX 7900 XTX, `STRATA_HIPBLASLT_TABLE=on` reads prompts of 4K tokens and more 1.33x to 1.89x as fast as the default (6 alternated pairs, CI95 within ±3%), decode is unchanged (1.009, CI95 [0.997, 1.021]), and the answers move far less than the R9700 table described in AMD_HIP.md: first-token KL 0.00036 mean over 37 prompts, the most likely token the same on 37 of 37, and 12 of 12 greedy 48-token answers identical. Given these results, could gfx1100 get the same "auto" decision as gfx1151 (2026-10-19), so that the table is used by default on this card? Everything below is the Linux package's hipBLASLt 100200; for 100500 see section 4. **Setup.** RX 7900 XTX 24 GB (gfx1100), Ryzen 7 8700G, 64 GB DDR5, Ubuntu 26.04.1, Linux 7.0.0. Strata 0.2.17 from `strata update` (engine/control 0.1.63, `hip-rx7900`, hipBLASLt 100200). Qwen3.8-Flash-Next GSQ-RCO IQ3_S, a launch preset with the setup defaults (262144 context, `--kv int8`, DFlash2 drafter, engine nice 5). The only difference between the two arms is `--set STRATA_HIPBLASLT_TABLE=on`; "off" leaves it unset, which is today's default. A fresh engine for every arm, the order alternated, nothing else running on the machine. The engine log shows the table in every "on" arm and in no "off" arm: ``` prefill gemm: hipBLASLt tuning enabled (26 rows, gfx1100, version 100200) ``` **1. Prompt speed and decode (package 0.2.17, hipBLASLt 100200), 6 pairs.** Cold prompts (unique from the first token, `cached_tokens` 0, `max_tokens` 1), prompt rate from the engine's own line (`N read in X ms`); lengths up to 32K run twice per arm. Decode: 6 prompts of 400 tokens at temperature 0. | | off | on | on/off | CI95 | | --- | ---: | ---: | ---: | --- | | prompt 1K (tok/s) | 336 | 338 | 1.004 | [1.000, 1.008] | | prompt 2K | 616 | 620 | 1.007 | [1.003, 1.011] | | prompt 4K | 829 | 1,104 | 1.333 | [1.327, 1.338] | | prompt 8K | 877 | 1,501 | 1.711 | [1.663, 1.759] | | prompt 16K | 950 | 1,752 | 1.845 | [1.820, 1.871] | | prompt 32K | 957 | 1,808 | 1.889 | [1.877, 1.900] | | prompt 64K | 955 | 1,796 | 1.880 | [1.864, 1.897] | | prompt 128K | 924 | 1,691 | 1.831 | [1.822, 1.839] | | decode (tok/s) | 62.0 | 62.6 | 1.009 | [0.997, 1.021] | **2. What the table does to the answers.** `STRATA_DUMP_FIRST_LOGITS`, compared with `terminal/tools/par/kl_logits.py`. 37 first windows: 24 prompts of 1.5K to 24K tokens built from the repository's docs, 6 short chat prompts (up to 63 tokens), 6 long prompts (4K and 12K) with 48 generated tokens, and the start-up warm-up request. Four arms: off, on, on again, off again. | | mean KL | median | max | top-1 same | | --- | ---: | ---: | ---: | ---: | | on vs off | 0.000356 | 0.000150 | 0.001749 | 37 / 37 | | off vs off (repeat) | 0.000019 | 0 | 0.000388 | 37 / 37 | | on vs on (repeat) | 0.000028 | 0 | 0.000565 | 37 / 37 | - The 24 long and 6 short prompts gave bit-identical logits when the same setting ran twice, so the on/off difference there is the table. The 6 prompts that generate 48 tokens differ slightly between two runs of the same setting too (max KL 0.0006), about as much as on against off. - Short prompts also show minor logit shifts (mean KL 0.0007), even though prefill speed is unchanged at these lengths. - Greedy 48-token answers (6 short, 6 long prompts): 12 of 12 identical between on and off, and between repeats. - For scale: AMD_HIP.md gives first-token KL 0.0157 mean, top-1 24 of 24, for the R9700 1.2.0 table, which is about 40 times what I measured here. **3. The standard suite and needles.** `strata bench run --url ... --skip-model-hash --track custom` (one run per arm; "nonconforming" only because the model hash was skipped to save time; both arms read the same model files), and `terminal/tools/needle_bench.py --lengths 32k,128k,250k --depths 10,50,90`. | | off | on | | --- | ---: | ---: | | prompt-4k tok/s (median of 3) | 886 | 1,215 | | prompt-16k tok/s | 975 | 1,860 | | TTFT 16K s | 16.83 | 8.85 | | decode short / 4k / 16k | 64.8 / 59.1 / 65.9 | 63.2 / 57.6 / 64.8 | | sanity checks | 12 / 12 | 12 / 12 | | needles (27.8K, 106K, 200K tokens x 3 depths) | 9 / 9 | 9 / 9 | | needle time, 200K tokens | 137 - 206 s | 80 - 116 s | The suite samples at temperature 1.0 with thinking on, so its decode column is one run each and only shows that nothing broke. **4. Source main and hipBLASLt 100500.** Built from main (3142b66b9), 3 alternated pairs each, on/off ratios: | | decode | 4K | 16K | 128K | | --- | ---: | ---: | ---: | ---: | | TheRock 7.10 (100200) | 0.994 | 1.335 | 1.827 | 1.829 | | TheRock 10.2.0a20261022 (100500) | 1.008 | 1.001 | 0.999 | 0.997 | Main matches the package. With 100500 the table loads but every row falls back (`STRATA_HIPBLASLT_VERBOSE=1`: `solution 2106 unavailable`, `solution 1034/1035/1036 rejects actual`), so speed and logits are unchanged. The table was calibrated on 10.2.0a20261003, a different nightly with the same version number, so the Windows `rdna3` package needs its own run. **What a change would touch.** The rule is in two places: `AUTO_ARCHS` in `terminal/strata_control/hipblaslt_tuning.py` and the `arch != "gfx1151"` check in `engine/src/prefill/gemm.cu` (`create_hipblaslt_state`), which the engine applies when it starts without the launcher. The engine uses a table only when the hipBLASLt version matches, and on the 100500 build above every row fell back, so the table had no effect there; the hipBLASLt of the Windows `rdna3` package is the one case I could not measure. I have not opened a pull request, since this is your decision; I can prepare one if it helps. Raw data (per-arm logs, engine-log slices, dumps' KL output, the scripts) is available if you want it. ### Reproduction status Measured locally
En el sitio
Enlaces a install, modelos, releases.