#6090 [Benchmark] RX 9070 XT (gfx1201) + DFlash2: two weeks >60 token/sec on Strata 0.2.10 / 0.1.56

closed · @fernandoracca · 3 comentários · No GitHub

BenchmarksServer & APIAMD / HIPModels & quantsDocumentationWindowsLinux

Descrição

A field report from a Linux gfx1201 box that your tables don't cover. Two
reasons for writing:

1. **Speed:** a 2019 Linux workstation + one RX 9070 XT + 64 GB DDR4 @ IQ3_S anchor has very good performance. still tuning it.
2. **Known problems:** your 0.2.14 notes carry *"Radeon gfx1201: DFlash2
   chosen by hand can stall on the prompt (11 of 50 engines on 0.2.11). Not
   looked at again in 0.2.14."* We have a long-running **positive** datapoint
   for that line, and the counterexample may help you find what broke.

## The machine

| | |
| --- | --- |
| CPU | AMD Ryzen 9 3900X (Zen 2, 2019) — the ~7-year-old part |
| RAM | 64 GiB (62.7 usable) |
| GPU | **one** RX 9070 XT 16 GiB (gfx1201) — the only new part in the box |
| Storage | NVMe (model + packs), experts served from RAM |
| OS | Ubuntu 26.10, kernel 7.3.0-10, GNOME Wayland |
| Strata | **0.2.10, Control + engine 0.1.56 (linux-x86_64-hip-rx9070), pinned** — deliberately: more on that below |

Model: Qwen3.8-Flash-Next, IQ3_S (ISTA-DASLab GSQ-RCO, 54.8 GB). Experts
(~50 GiB) pinned in RAM, hot experts cached in VRAM, PLE/n-gram streamed from
SSD. Shape: `--kv k8v4 --kv-resident 32768 --max-context 196608
--vram-reserve-mib 700 --expert-cache auto --prefill auto --draft dflash2
--draft-model <q4_0-selq4.pack> --draft-fallback none --draft-conf 0.7`
(windows up to 8 tokens, spec 8). Boot shape: **4,949 expert slots, 862 MiB
VRAM free**; parking off (`--conversation-cache-mib 0`) — deep sessions
re-prefill instead, which prefill speed makes affordable.

## Controlled bench (2026-11-03, the pinned build, drafter ladder)

`strata bench run --suite standard-v1`, ×2 paired runs per arm, fresh engine
session per run, decode deltas <10% treated as noise; long-answer probe =
3×~8K-token generations with model-card sampling and thinking on; deep probe
= ~130K-token synthetic prompt with a needle. Engine 0.1.56 throughout.

| arm | decode short/4K/16K (tok/s) | long-answer band (tok/s) | prompt 4K/16K (tok/s) | TTFT short | slots |
| --- | --- | --- | --- | --- | --- |
| no drafter | 49.1 / 48.8 / 49.2 | 52–55 | 1,489 / 2,379 | 0.68 s | 5,361 |
| MTP | 68.1 / 60.9 / 62.1 | 63.9 | 1,242 / 2,189 | 0.71 s | 4,817 |
| **DFlash2 (selq4)** | **69.7 / 61.1 / 62.9** | **74.3** (66.8 excl. one 89.5 outlier) | 1,450 / 2,355 | 0.71 s | **4,949** |

Deep 130K probe on the DFlash2 arm: prefill **2,199 tok/s**, decode 35.9,
**min card free 262 MiB** (best of the whole sweep), needle ok, zero watchdog
events. Expert cache hit ~0.80; DFlash2 boots *more* slots than MTP at the
same reserve (the selq4 pack costs 380 MiB measured vs the MTP layer's
1.08 GiB) — the drafter funds itself, exactly as your `CONF_STEPS` ladder
assumes.

DFlash2 ≥ MTP in every decode cell; the long band (the one agent traffic
actually lives in) is where it wins biggest: 74.3 vs 63.9.

## Daily production on gfx1201 + DFlash2 (the counterexample)

DFlash2 has been the live drafter on this card since 2026-11-03, serving an
agentic coding workload (long sessions, 100K–150K prompt depths, prompt
cache doing most of the re-reading).

- **123 engine boots on 0.1.56 with DFlash2: zero prompt stalls.** The
  0.2.11 failure mode ("can stall on the prompt", 11/50) has never appeared
  on 0.1.56, across ~2 weeks and daily restarts (flavor flips, reboots, the
  blue-swap migration).
- One engine death in the period (11-04): a single `verify: timed out at
  layer 0 … (#267)` followed by a GPU wedge that outlived the process (two
  silent hung boots until reboot). One occurrence in 134 boots; not a prompt
  stall, and the drafter was not implicated by your #267 history either.
- A separate incident (11-05): suspend/resume with the engine holding ~15 GiB
  pinned VRAM + ~50 GiB userptr produced an `sq_intr` fault → driver GPU
  reset killed every GPU client. Now automated away with
  sleep/wake systemd units (stop before sleep, start after resume). Worth a
  docs note for anyone running Strata on a laptop: **suspend with the engine
  resident is the most fragile path on this box.**
- Live acceptance, current soak slot: **194,104 of 261,699 draft tokens
  accepted = 74.2%** over 448 serve requests (engine log, greppable).
- Expert cache on this shape (cumulative soak log): **82.1% hit rate**
  (189.3M/230.6M lookups); only **5.8% of 244.9M routed lookups** miss the
  VRAM cache, and those stream from RAM over PCIe (userptr), not SSD — live
  decode reads **0.04 MB/s from disk** (PLE stream sits in page cache). The
  16 GiB card is a cache-tier player here, not the memory wall.
- Field record output rate: **112.0 tok/s** (244 tokens generated on a warm
  short prompt; re-run observed **117.0 tok/s** live on 2026-11-16 — that line
  flushes to the log at request completion, quote the logged 112.0 until then);
  typical agent decode 50–66 tok/s at 100K+ depth; one
  observed prefill of **2,281 tok/s** on a boot where `--prefill auto` chose
  chunk 16384 (the boot-adaptive chunk varies 8192↔16384 — pin
  `STRATA_GFX12_CHUNK=16` for strict comparisons).
- Soak stability on the current slot (since 11-14): `NRestarts=0`, zero
  journal errors, zero fault-space lines (gpu reset / sq_intr / driver
  paused). Your R9700-Linux "driver paused 115–810 ms" known-problem has no
  counterpart on this gfx1201 box.

**The ask:** whatever changed between 0.1.56 and 0.1.57 that made DFlash2
stall on the prompt on gfx1201 is invisible to us — 0.2.12+ defaults gfx1201
Auto to MTP, and no release through 0.2.14 names a fix. We deliberately froze
at 0.1.56 rather than ship our users a stall. If you ever name a fix, we will
test it in our green slot against the pinned blue and report — and if it
helps, our whole ledger re-baselines. Happy to share the arms ledger
(per-arm standard-v1 reports, boot arithmetic, VRAM curves) as CSV if useful
for a gfx1201 row in your tables.

## Update, 2026-11-17: ran the newer engine in the green slot, and the stall is still there

We held the promise above and tested the newer line in a disposable second
slot (engine **0.1.60** / Control **0.1.61**, same box, same DFlash2
`q4_0-selq4.pack` chosen by hand, `--spec 6 --draft-conf 0.7`, `--kv k8v4
--kv-resident 32768 --kv-grow`, `--max-context 262144`, `--vram-reserve-mib
700`), with the production slot never touched:

- **1 prompt stall / 41 cold ≥12K prefills on 0.1.60.** 40 scripted cold 16K
  soaks completed (median 1,770 tok/s, 11 of them below 1,000); the 41st, the first 16K request of
  `standard-v1` on that boot, stalled at
  `reading the prompt (batched): layer 25 of the prompt chunk from token 0`,
  the watchdog stopped the engine and it **aborted with exit −6** — a harsher
  outcome than 0.1.56, where the same event only self-restarts. The other 20
  bench requests then completed and sanity passed 12/12.
- For comparison on the same box: **1 / 380** cold 16K prefills on 0.1.56
  (test slot), **0 / 39** on 0.1.56 real agent traffic (production slot).
- Speed on 0.1.60 did not show up as something we can claim: prompt-16K
  2,351.9 tok/s (n=2, one request lost) against our 0.1.56 anchor's 1,991.7,
  decode 65.4 / 57.8 / 57.3 against 66.7 / 60.5 / 61.1 — inside noise, and
  the same build read a cold-16K median of 1,770 in one boot and 2,353 in the
  next. **Our per-boot cold-prefill medians on identical shapes span
  1,049–2,353 tok/s across 18 boots**, which is the number we now check any
  claim against.
- Full report with the stall report, the ruled-out list and the negative
  result on the desktop-rendering trigger from #579: our comment on **#579**;
  the family datapoint also sits on **#3807** and **#5892**.

Conclusion from our side: **still pinned at 0.1.56**, because the upgrade did
not remove the prompt stall and turned it into an abort.

We have seen multiple max decodes reaching 117 tokens/sec, even at 192K
context. I intend to grow it to 256K context if I can get it stable enough —
the 262144 shape is live on the production slot now, and the one thing
stopping it from being permanent is the stall above.

Happy to answer with numbers. — tungsten (single-user Linux lab box, gfx1201,
running 0.2.10 pinned by tag + snapshot, feed seq 120 frozen by choice).

strata 0.2.10 (build 0.1.56; python 3.12.15)

No site

Links install, modelos, releases.