#6531 [Benchmark] gfx1100 (RX 7900 XTX, IQ3_S): existing switches give 1.9–3.7x prompt and 1.2x decode speedup

open · @superbaenawa · 0 comentários · No GitHub

BenchmarksSetup & installMulti-GPUAMD / HIPModels & quantsDocumentationLinux

Descrição

### What did you measure, or what went wrong

**Summary.** On an RX 7900 XTX, the package with its setup defaults reads prompts at 336 to 950 tok/s and decodes at 62.4 tok/s. With a set of switches that already exist in the engine (no code change; list in section 1), the same package reads prompts at 792 to 3,359 tok/s (1.9x to 3.7x; 16K and longer 2.8x to 3.7x) and decodes at 75.7 tok/s (1.21x). Time to first token for a 16K prompt drops from 15.5 s to 5.5 s. Against the setup defaults, first-token KL over 36 requests is 0.00035 mean, the most likely token is the same on 36 of 36, and every greedy answer that differs says the same thing in other words. gfx1100 has no entry in the per-architecture defaults today, so every one of these has to be set by hand. Could gfx1100 get defaults along these lines, or the subset you are comfortable with? #6429 already covers the hipBLASLt table, which is one of them.

**Setup.** RX 7900 XTX 24 GB (gfx1100), Ryzen 7 8700G (8 cores), 64 GB DDR5, Ubuntu 26.04.1, Linux 7.0.0. Strata 0.2.17 from `strata update` (engine/control 0.1.63, `hip-rx7900`, hipBLASLt 100200). Qwen3.8-Flash-Next GSQ-RCO IQ3_S, 262144 context, `--kv int8`, DFlash2 drafter (`--spec 6 --draft-conf 0.7`), as the launch preset sets them. A fresh engine for every run, nothing else running.

**1. The set.** Launch arguments `--kv-resident 32768 --pcie-frac 0 --ple-io direct --adapt-every 1`, and:

```
STRATA_ENGINE_NICE=0
STRATA_HIPBLASLT_TABLE=on          STRATA_PF_GEMM=1            STRATA_PF_F16=1
STRATA_HIP_WMMA=1                  STRATA_SELECT_WMMA=1        STRATA_QFUSE=1
STRATA_PF_FUSED=1                  STRATA_PF_PAD=1             STRATA_HC_UPMIX=1
STRATA_PF_HCDOWN=1                 STRATA_PF_HC_REGION=1       STRATA_EMB_REUSE_ACCOUNT=1
STRATA_GDN_HEAD=1                  STRATA_GDN_CHUNKED=1        STRATA_GDN_PP=1
STRATA_GDN_CONVL2=1                STRATA_GDN_NOY=1            STRATA_PA_FAST=1
STRATA_PREFILL_RING=192            STRATA_STAGER_RING=64       STRATA_KV_HOST_DMA=1
STRATA_PREFILL_CPU_SHARE=0.6       STRATA_PREFILL_CPU_SHARE_MAX=10240
STRATA_PREFILL_AUTO_MAX=40960      STRATA_ADAPT_OVERLAP=1      STRATA_QSA_ROWS=1
STRATA_VERIFY_QDEDUP=1             STRATA_FUSE_CONVAB=1        STRATA_HEAD_MIX_MULTI=1
STRATA_SH_STREAM=0                 GPU_MAX_HW_QUEUES=6
```

**2. Speed, setup defaults against the set.** Same order as for the defaults: fresh engine, `decode.py` (6 prompts of 400 tokens, temperature 0), then short-question decode, two queued requests, a conversation, then cold prompts of 1K to 128K (unique from the first token, `cached_tokens` 0, `max_tokens` 1). Prompt rate is the engine's own `N read in X ms`. The set: 5 rounds. The defaults: 5 rounds for the decode rows, 1 round for the prompt sweep.

| | defaults | the set | ratio |
| --- | ---: | ---: | ---: |
| decode, 6 prompts (tok/s) | 62.4 | 75.7 | 1.21 |
| decode, short questions | 60.9 | 75.1 | 1.23 |
| two queued requests, aggregate | 55.4 | 68.6 | 1.24 |
| decode after a 22-24K prompt | 66.1 | 82.6 | 1.25 |
| TTFT 16K / 23.5K (s) | 15.5 / 24.0 | 5.5 / 7.2 | |
| prompt 1K (tok/s) | 336 | 792 | 2.36 |
| prompt 2K | 583 | 1,118 | 1.92 |
| prompt 4K | 827 | 1,554 | 1.88 |
| prompt 8K | 899 | 1,976 | 2.20 |
| prompt 16K | 936 | 2,625 | 2.80 |
| prompt 32K | 947 | 2,819 | 2.98 |
| prompt 64K | 950 | 3,300 | 3.47 |
| prompt 128K | 911 | 3,359 | 3.69 |
| draft acceptance | 83.6% | 83.3% | |

**3. What the set does to the answers.** `STRATA_DUMP_FIRST_LOGITS` on 36 fixed requests: 24 prompts of 1.5K to 27K tokens from the repository's docs (`max_tokens` 1), 6 short chat prompts and 6 long prompts (4K and 12K) with 48 greedy tokens. Draft costs fixed for these runs (`STRATA_SPEC_FIXED=1`, `STRATA_SPEC_FIXED_TABLE=28,30,33,37,47,50,57,66`): with the live costs, which are the HIP default, the window width changes between starts and so do the logits of the generated requests. With fixed costs, the same setting run twice gives 36 of 36 identical dumps and 12 of 12 identical answers.

| | mean KL | max KL | top-1 same | greedy answers identical |
| --- | ---: | ---: | ---: | ---: |
| the set vs defaults | 0.000348 | 0.00285 | 36 / 36 | 4 / 12 |
| the set, run twice | 0 | 0 | 36 / 36 | 12 / 12 |

I read all 8 greedy pairs that differ. They are the same answers in other words (for example "Need provide one sentence" against "Need produce one sentence"), with no wrong answer and none off topic. The launcher's readiness request ("hi", 53 tokens with the template) is left out of the 36: its KL is 0.0227 with the same top-1, and it comes from `--kv-resident 32768` and `--pcie-frac 0`, which move only the first request after start; the requests after it are in the same range as the rest.

**4. Which switches carry it.** Each one was checked on its own, either taken out of the set or added to it (the share values against the ones I had set before, not against the engine's automatic share), and kept only if its first-token KL stayed at or below 0.001 mean with the same top-1 on 36 of 36 and no wrong greedy answer. Speed was measured per switch where it was large enough to see (3 to 5 alternated rounds; CI95 from the pairs):

| switch | effect on its own |
| --- | --- |
| `STRATA_HIPBLASLT_TABLE=on` | prompts ≥ 4K 1.33x to 1.89x (#6429) |
| `PF_PAD`, `PA_FAST`, `GDN_HEAD`, `STAGER_RING`, `PREFILL_RING`, `KV_HOST_DMA` (taken out together) | without them, prompts ≥ 8K are 12% to 18% slower |
| `STRATA_PF_GEMM=1` | prompts ≥ 16K +3% to +5% |
| `STRATA_PF_F16=1` | prompts ≥ 16K +3% to +4% |
| `STRATA_PREFILL_CPU_SHARE=0.6` (against 0.3) | 1K +22%, 2K +14% |
| `STRATA_PREFILL_CPU_SHARE_MAX=10240` | 8192 against 4096: 4K +30%, 6K +10%; 10240 against 8192: 8.3K and 9.1K +10%, 12K −0.5% |
| `STRATA_PF_HC_REGION=1` + `STRATA_EMB_REUSE_ACCOUNT=1` | 1K to 128K +1.2% to +3.9% (5 rounds; the ratio's CI95 lies above 1.0 at 32K to 128K) |
| `--pcie-frac 0` | decode +8% (74.4 against 68.9) |

The rest were measured only in steps, 5 rounds each: with the hipBLASLt table on, the four launch arguments and `ENGINE_NICE=0` raised decode from 62.7 to 72.8 tok/s; adding the env switches of section 1 as they stood then (all of them except `PF_F16`, `PF_HC_REGION` and `EMB_REUSE_ACCOUNT`, with the CPU share at 0.3 and its limit at 4096) raised prompts above 10K from 1,745 to 2,753 tok/s. I can't say what each switch in that step adds on its own.

On the CPU share limit: at 8192 the step is sharp. 8,190-token prompts read at 2,229 tok/s and 8,197-token prompts at 2,043 tok/s (−8%, 3 rounds, exact token counts). Raising the limit to 10240 moves that step past the range where the share still helps; 12288 was 5.7% slower at 12K.

Two of the switches change the bits on this card although the comment near the code says they keep them: `STRATA_PF_HC_REGION=1` ("The same kernels on the same values: the same bits", `prefill.cpp`) and `STRATA_PA_FAST=1` ("the same bits", measured on gfx1151, `qsa_prompt_attn.cu`). Both stay within the limits above (mean KL 0.000088 and 0.000262).

**What a change would touch.** The per-architecture table in `engine/src/core/arch_defaults.cpp` has no gfx1100 entry, and the launcher's tuning presets don't set these for gfx1100 either. Which of these belong in engine defaults and which in the launcher is your call; I can prepare a pull request for whatever subset you choose. The CPU share values are tuned on an 8-core CPU and may want a different value on other machines. Likewise `--kv-resident 32768` and `--pcie-frac 0` fit this card's 24 GB with this model (IQ3_S, 262144 context, int8 KV); I have not tried them on a card with less memory, such as the 20 GB RX 7900 XT, or with a larger model, so they may need to scale with free VRAM rather than be fixed values.

Raw data (per-run logs, engine-log slices, the KL and greedy comparisons, the scripts) is available if you want it.

### Reproduction status

Measured locally

No site

Links install, modelos, releases.