#5762 [Benchmark] DFlash2 on gfx1201 (R9700, 32 GB): no prompt stall on 0.1.59.1; window 8 is what pays, conf 0.35 costs 10%
closed · @djeddyau · 2 commentaires · Sur GitHub
BenchmarksMulti-GPUAMD / HIPNVIDIA / CUDAModels & quantsDocumentation
Description
# [Benchmark] DFlash2 on gfx1201 (R9700, 32 GB): no prompt stall on 0.1.59.1; window 8 is what pays, conf 0.35 costs 10%
All numbers below were produced on the machine described in the Environment section, on
2026-11-11, with `tools/strata_bench.py` and the engine built from this checkout. No number here
comes from another machine or from a model run.
**Result in one line:** on a 32 GB RDNA4 card the DFlash2 prompt stall does not appear, the
drafter's window (`--spec`) is the lever that matters and not the confidence gate, and the
confidence ladder's low rungs cost about 10% on a card that is not streaming experts.
## Why I am opening this instead of commenting on an existing thread
I searched for `dflash2` (122 issues) and for `gfx1201` + stall. The closest threads are #5687
(DFlash2 for this model, but UD-IQ3_XXS on an RTX 5070), #5720 (calibration sweeps, stock pack),
#5614 (gfx1201, 0.1.58, `--spec 4 --mtp` fails to launch) and the closed #5237 (drafter matrix
on an RTX 5080). None of them is this card with a 32 GB pack and Auto drafting.
#5549 (`[Benchmark] Single vs dual rx9070 (hip): engine 0.1.58, ud-iq4_xs-v31`) is the nearest
hardware match and would have been the natural home for some of this, but the page would not
load for me (HTTP 403), so **I cannot say this is not a duplicate of #5549** — please redirect
it if it is.
This issue changes no default and asks for none. It reports three measurements against code that
is in `main`.
## Environment
- GPU: AMD Radeon AI PRO R9700 (gfx1201 / RDNA4), 34,208,743,424 B VRAM; driver/amdgpu
7.1.9.31600000
- CPU: AMD Ryzen 9 9900X 12-Core (24 threads); 64 GB RAM
- OS: Ubuntu 26.04.1 LTS, kernel 7.0.0-43-generic
- Engine: 0.1.59.1, HIP build, HIP version 7.2.26015, built locally on 2026-11-10 from
`dbb2d168cffa8cf793d9325358809d5263c7c6b0`
- Model: `unsloth/Qwen3.8-Flash-Next` UD-IQ4_XS-v31, pack `unsloth-ud-iq4_xs-v31`;
262,144 context, `--kv int8 --kv-resident 32768`, `--resident-budget-gib 37`,
`--vram-reserve-mib 2048`, `--vision`, `--spec-min-p 0.5`, `--expert-profile`
- Drafter packs, both under `Strata-data/dflash2/`:
- `qwen3.8-flash-next-dflash2.q4_0.pack` — 522,458,112 B, sha256
`cda40d62d0eb8f37ed6e272142641a1d2f0de4a32f9e14d0416e7a71534849f1`
(downloaded from `incoai/Qwen3.8-Flash-Next-DFlash2` at `a22a1744`; the repo's own sha256
`005377b4d8e07e06787aee095b4b58b8497051e72807655f82a90b81205396a8` is for `model.safetensors`)
- `qwen3.8-flash-next-dflash2.q4_0-selq4.pack` — 339,694,592 B, sha256
`4a42b98220ea7880664002c8edda1563aecbae754b762a2e2f31b2b91ee93ecd`, "selector codebooks: Q4_0
rows", made locally by `strata_control.drafting_fetch.convert_selector` (this is what
`apply()` picks when the file is present — **every measurement below used the selector pack,
not the stock pack**)
- `draft_vocab.bin` — 425,196 B, sha256 `b1e1d3a7a9e4bf862dcd5923ce661fb59bbd07907e594df5cf86a62ac235cb91`
## What was run
Arms differ **only** in the drafter and its knobs. Every arm pinned `--expert-cache 7800`, which
gives **9,117 expert slots on every arm** (`expert_slots` confirmed in each result file). This
matters: the auto rule gives a DFlash2 start about 117 more slots than an MTP start because the
drafter is smaller than the MTP layer, and letting one arm buy expert slots with VRAM the other
arm spends on its own drafter is exactly the unfairness #5720 raises.
```sh
python3 strata.py drafting fetch --config strata-unsloth-ud-iq4_xs-v31.json --mode dflash2
# (mode dflash2 is required on this card: Auto is mtp here, and the fetch honours Auto)
# then convert_selector by hand, because the fetch leaves the q4_0 pack as it is and apply()
# wants the selector pack:
python3 -c "from strata_control import drafting_fetch as f; \
f.convert_selector('.../dflash2/qwen3.8-flash-next-dflash2.q4_0.pack')"
.venv/bin/python tools/strata_bench.py --config <arm>.json --port 28818 --runs 3
```
Suite: `tools/strata_bench.py` default suite, runs = 3, thinking off, temperature 0. Arms were
run one after another by a detached script that stopped the production server first and always
restored it. The server's own start log shows which drafter and window each arm used.
## Smoke test: the prompt stall did not reproduce
Engine 0.1.59.1, DFlash2 at the flags `apply()` picks (`--draft dflash2 ... --spec 6`,
confidence unset → 0.5), auto expert cache (10,315 slots):
```
DFlash2 drafter on ...: no MTP layer is loaded; the drafter is the only guesser
DFlash2: 1206-token prompt at 999 tok/s; 256 tokens at 79.8 tok/s, 150/186 drafts accepted
DFlash2: 180-token prompt at 1137 tok/s; 128 tokens at 70.4 tok/s, 46/56 drafts accepted
```
One clean start, no `prompt stalled` line, no `--draft-fallback none` trip, prompt speed in the
usual band for this card. I ran two prompts, not the fifty engines that produced the 11/50 figure
behind the `AUTO_MTP` comment, so treat this as "did not reproduce once" and not as "fixed".
Memory in the same start: **DFlash2 492 MiB pack + 81.2 MiB head = 573 MiB against the MTP
layer's 835 MiB** — 262 MiB that the auto rule spends on 117 extra expert slots.
## Sweep: eight arms, one card, same slots
| arm | chat tok/s | agent tok/s | drafts accepted |
|---|---:|---:|---:|
| mtp-base (`--spec 4 --mtp`) | 70.8 | 80.3 | 61.0% |
| **mtp-base again, identical config** | **67.9** | **78.7** | 60.2% |
| mtp + `STRATA_SPEC_COUPLED=1` | 70.7 | 73.0 | 59.1% |
| df2 conf 0.35 | 63.3 | 81.3 | 54.0% |
| df2 conf 0.5 (what the code picks) | 72.9 | 77.6 | 61.8% |
| df2 conf 0.65 | 70.7 | 76.9 | 71.9% |
| df2 conf 0.80 | 71.1 | 81.1 | 81.7% |
| df2 conf 0.5 + `--spec 8` | 74.5 | 78.6 | 63.5% |
The second row is the same arm as the first. It moved chat 4.1% and the agent case 2.0%. Any
single-run gap smaller than that is noise, and most of the first sweep is smaller than that.
For scale, the same model and drafter as a long-running production server, measured from its
own `/metrics` over 52 real agent requests at 10,198 slots: 74.7 tok/s mean, 16,332 of 24,070
drafts accepted (67.9%). Absolute numbers above are a little lower because 9,117 slots is less
than auto's ~10.2k, which is the price of equal footing.
## The top of the ladder, then five interleaved pairs
| arm | chat tok/s | agent tok/s | accepted |
|---|---:|---:|---:|
| df2 conf 0.90 | 63.9 | 75.5 | 84.3% |
| df2 conf 0.95 | 61.5 | 73.5 | 89.1% |
| df2 conf 0.80 + `--spec 8` | 71.3 | 82.6 | 81.0% |
The ladder turns over between 0.8 and 0.9: acceptance keeps climbing because a higher gate means
fewer, safer drafts, and speed falls with them. 0.8 is near the top for this card, not the bottom
of an untested range.
Paired A/B, 5 interleaved pairs, order alternated, the best chat arm from the sweep
(conf 0.5 + window 8) against the MTP, 9,117 slots both sides:
| pair | 1 | 2 | 3 | 4 | 5 | median |
|---|---:|---:|---:|---:|---:|---:|
| chat Δ | +2.4 | +0.9 | +0.8 | +5.0 | +3.2 | **+2.4 (+3.4%), 5/5 pairs faster** |
| agent Δ | −0.4 | −6.6 | +1.2 | −0.5 | +1.0 | −0.4 (−0.5%), 2/5 faster |
Short chat decode: faster in every pair (a sign test on five pairs is p = 1/16 for either side,
so "5/5" is suggestive, not proof). The long-context agent case, which is what core19 measures:
a wash.
## Three things this says about the code, and one question
1. **`AUTO_MTP`'s reason for gfx1201** — "the DFlash2 prompt stall (11 of 50 engines on 0.2.11),
not the slot rule" — did not reproduce once on 0.1.59.1. Worth re-testing the entry before the
next release closes it out; I do not have enough runs to call it fixed.
2. **A DFlash2 chosen by name on RDNA4 runs at `--spec 6`.** `dflash2_auto_cap`
(`include/strata/core/draft_cap.hpp`) returns no cap for RDNA4, so `--spec 8` is reachable —
and window 8 is the only lever in this whole exercise that produced a reproducible gain
(window 6 at the same confidence: no gain over the MTP). If someone opts in with
`"drafting": "dflash2"` on a 32 GB RDNA4 card, is 6 the intended window, or is it just
what `SET_DEFAULTS` happens to hold?
3. **`CONF_STEPS`' low rungs are for cards that stream experts.** conf 0.35 costs 10% chat on a
card holding 9,117 slots. The auto pick (0.5) is fine here, and 0.8 is close to the top. The
same ladder on a 12 GB card at 300 slots may well be right; I cannot test that.
4. Question, not a finding: the drafter is 262 MiB smaller than the MTP layer. Is the ~117 slots
that difference buys expected to be worth more than the difference in acceptance behaviour?
It is the one confound that any drafter comparison on this card has to pin, and pinning it is
what made the first sweep flat.
## What I did not run
- `strata bench run` — the standard protocol (thinking on, warmup, two alternating arms, the
three medians in docs/COMMUNITY_BENCHMARKS.md). Everything above is the `strata_bench.py`
micro-suite with thinking off and a pinned expert cache, so **the absolute tok/s values here are
not comparable with standard-protocol submissions**.
- Terminal-Bench core19 (the accuracy side; no correctness check of any kind was run).
- `--batch`, `--slots`/`--replicas`, `--parallel`, the 32K prompt test, and any other GPU,
backend (CUDA, SYCL), OS or model. Nothing here applies to another card.
- The 50-engine stall protocol from #5237.
- No default was changed. The production server on this box stayed on the MTP (`--spec 4 --mtp`)
throughout, and its config file was never edited — each arm ran from a temporary copy.
- `STRATA_IQ_MT_MIN=1` was not set, so IQ decode is not byte-reproducible between these runs;
that is part of why the identical-arm repeat moved as much as it did.
## Evidence
Every run's raw `results.json`, README, engine log and the runner log (with the start flags and
`/metrics` for each arm) are on the machine that produced them. The runner script is a 200-line
Python driver that generates the arms from one config; I can post it, or the per-arm config
diffs, or the raw files, on request. Nothing in this issue was copied from another report.
[df2-evidence-gfx1201.tar.gz](https://github.com/user-attachments/files/34669319/df2-evidence-gfx1201.tar.gz)Sur le site
Liens install, modèles, releases.