#6429 [Benchmark] hipBLASLt tuning table by default on gfx1100 (RX 7900 XTX, IQ3_S): speed, decode and logits

open · @superbaenawa · 4 コメント · GitHub で見る

BenchmarksSetup & installAMD / HIPModels & quantsDocumentationWindowsLinux

本文

### What did you measure, or what went wrong

#5666 reported the speed on 0.2.13 and was closed as a benchmark record, without a decision on the default. This asks for that decision, with the decode and answer measurements it depends on.

**Summary.** On an RX 7900 XTX, `STRATA_HIPBLASLT_TABLE=on` reads prompts of 4K tokens and more 1.33x to 1.89x as fast as the default (6 alternated pairs, CI95 within ±3%), decode is unchanged (1.009, CI95 [0.997, 1.021]), and the answers move far less than the R9700 table described in AMD_HIP.md: first-token KL 0.00036 mean over 37 prompts, the most likely token the same on 37 of 37, and 12 of 12 greedy 48-token answers identical. Given these results, could gfx1100 get the same "auto" decision as gfx1151 (2026-10-19), so that the table is used by default on this card? Everything below is the Linux package's hipBLASLt 100200; for 100500 see section 4.

**Setup.** RX 7900 XTX 24 GB (gfx1100), Ryzen 7 8700G, 64 GB DDR5, Ubuntu 26.04.1, Linux 7.0.0. Strata 0.2.17 from `strata update` (engine/control 0.1.63, `hip-rx7900`, hipBLASLt 100200). Qwen3.8-Flash-Next GSQ-RCO IQ3_S, a launch preset with the setup defaults (262144 context, `--kv int8`, DFlash2 drafter, engine nice 5). The only difference between the two arms is `--set STRATA_HIPBLASLT_TABLE=on`; "off" leaves it unset, which is today's default. A fresh engine for every arm, the order alternated, nothing else running on the machine.

The engine log shows the table in every "on" arm and in no "off" arm:

```
prefill gemm: hipBLASLt tuning enabled (26 rows, gfx1100, version 100200)
```

**1. Prompt speed and decode (package 0.2.17, hipBLASLt 100200), 6 pairs.** Cold prompts (unique from the first token, `cached_tokens` 0, `max_tokens` 1), prompt rate from the engine's own line (`N read in X ms`); lengths up to 32K run twice per arm. Decode: 6 prompts of 400 tokens at temperature 0.

| | off | on | on/off | CI95 |
| --- | ---: | ---: | ---: | --- |
| prompt 1K (tok/s) | 336 | 338 | 1.004 | [1.000, 1.008] |
| prompt 2K | 616 | 620 | 1.007 | [1.003, 1.011] |
| prompt 4K | 829 | 1,104 | 1.333 | [1.327, 1.338] |
| prompt 8K | 877 | 1,501 | 1.711 | [1.663, 1.759] |
| prompt 16K | 950 | 1,752 | 1.845 | [1.820, 1.871] |
| prompt 32K | 957 | 1,808 | 1.889 | [1.877, 1.900] |
| prompt 64K | 955 | 1,796 | 1.880 | [1.864, 1.897] |
| prompt 128K | 924 | 1,691 | 1.831 | [1.822, 1.839] |
| decode (tok/s) | 62.0 | 62.6 | 1.009 | [0.997, 1.021] |

**2. What the table does to the answers.** `STRATA_DUMP_FIRST_LOGITS`, compared with `terminal/tools/par/kl_logits.py`. 37 first windows: 24 prompts of 1.5K to 24K tokens built from the repository's docs, 6 short chat prompts (up to 63 tokens), 6 long prompts (4K and 12K) with 48 generated tokens, and the start-up warm-up request. Four arms: off, on, on again, off again.

| | mean KL | median | max | top-1 same |
| --- | ---: | ---: | ---: | ---: |
| on vs off | 0.000356 | 0.000150 | 0.001749 | 37 / 37 |
| off vs off (repeat) | 0.000019 | 0 | 0.000388 | 37 / 37 |
| on vs on (repeat) | 0.000028 | 0 | 0.000565 | 37 / 37 |

- The 24 long and 6 short prompts gave bit-identical logits when the same setting ran twice, so the on/off difference there is the table. The 6 prompts that generate 48 tokens differ slightly between two runs of the same setting too (max KL 0.0006), about as much as on against off.
- Short prompts also show minor logit shifts (mean KL 0.0007), even though prefill speed is unchanged at these lengths.
- Greedy 48-token answers (6 short, 6 long prompts): 12 of 12 identical between on and off, and between repeats.
- For scale: AMD_HIP.md gives first-token KL 0.0157 mean, top-1 24 of 24, for the R9700 1.2.0 table, which is about 40 times what I measured here.

**3. The standard suite and needles.** `strata bench run --url ... --skip-model-hash --track custom` (one run per arm; "nonconforming" only because the model hash was skipped to save time; both arms read the same model files), and `terminal/tools/needle_bench.py --lengths 32k,128k,250k --depths 10,50,90`.

| | off | on |
| --- | ---: | ---: |
| prompt-4k tok/s (median of 3) | 886 | 1,215 |
| prompt-16k tok/s | 975 | 1,860 |
| TTFT 16K s | 16.83 | 8.85 |
| decode short / 4k / 16k | 64.8 / 59.1 / 65.9 | 63.2 / 57.6 / 64.8 |
| sanity checks | 12 / 12 | 12 / 12 |
| needles (27.8K, 106K, 200K tokens x 3 depths) | 9 / 9 | 9 / 9 |
| needle time, 200K tokens | 137 - 206 s | 80 - 116 s |

The suite samples at temperature 1.0 with thinking on, so its decode column is one run each and only shows that nothing broke.

**4. Source main and hipBLASLt 100500.** Built from main (3142b66b9), 3 alternated pairs each, on/off ratios:

| | decode | 4K | 16K | 128K |
| --- | ---: | ---: | ---: | ---: |
| TheRock 7.10 (100200) | 0.994 | 1.335 | 1.827 | 1.829 |
| TheRock 10.2.0a20261022 (100500) | 1.008 | 1.001 | 0.999 | 0.997 |

Main matches the package. With 100500 the table loads but every row falls back (`STRATA_HIPBLASLT_VERBOSE=1`: `solution 2106 unavailable`, `solution 1034/1035/1036 rejects actual`), so speed and logits are unchanged. The table was calibrated on 10.2.0a20261003, a different nightly with the same version number, so the Windows `rdna3` package needs its own run.

**What a change would touch.** The rule is in two places: `AUTO_ARCHS` in `terminal/strata_control/hipblaslt_tuning.py` and the `arch != "gfx1151"` check in `engine/src/prefill/gemm.cu` (`create_hipblaslt_state`), which the engine applies when it starts without the launcher. The engine uses a table only when the hipBLASLt version matches, and on the 100500 build above every row fell back, so the table had no effect there; the hipBLASLt of the Windows `rdna3` package is the one case I could not measure. I have not opened a pull request, since this is your decision; I can prepare one if it helps.

Raw data (per-arm logs, engine-log slices, dumps' KL output, the scripts) is available if you want it.

### Reproduction status

Measured locally

関連リンク

インストール・モデル・リリースへのサイト内リンク。