#2124 Community Benchmark: Multi-GPU Nvidia, Q3-XXS, Q3-S, Q4-XS comparison

Description

# 2026-10-12: 4x NVIDIA (RTX 5060 Ti + 2x RTX 3090 + RTX 3060), Xeon W-1350, 123 GiB RAM — three quants of Qwen3.8-Flash-Next on one box

**Summary: across IQ3_XXS → IQ3_S → UD-IQ4_XS, decode barely moves (111 / 103 / 91 tok/s) while prompt processing
collapses (3,228 / 1,946 / 1,270 tok/s at 32K, TTFT 9.96 → 16.50 → 25.28 s). The ordering tracks how many experts the
engine could make *resident* — 98% / 86% / 81%, 24,387 / 21,537 / 20,132 of 24,576 profiled pairs — not the quant name,
and it is super-linear: 12 points of residency costs 40% of prefill, 17 points costs 61%. On a multi-GPU box where
three of four links probe 3.1 GB/s, the quant choice is a prefill decision. I kept IQ3_XXS.**

Speed only: no correctness suite was run on any arm.

## Machine (identical for all three arms)

- **GPUs:** RTX 5060 Ti 16 GB + RTX 3090 24 GB + RTX 3090 24 GB + RTX 3060 12 GB = 76 GiB VRAM, all four selected
- **PCIe — the number that drives everything here:** the 5060 Ti probes **13.9 GB/s** (x8); both 3090s and the 3060 probe
  **3.1 GB/s** (an Oculink/x4 riser and a x4-shaped slot). The engine logs `pcie_frac 0.09` for those three and prices
  an expert miss at **x6.35**.
- **Power:** both 3090s capped at 225 W, the 3060 at 125 W (the cap binds during generation).
- **CPU / RAM / disk:** Xeon W-1350 @ 3.30 GHz (6C/12T), 123.3 GiB RAM, NVMe (1.9 TB, 484 GB free).
- Linux 7.0.0-38-generic, driver 595.91.07, CUDA 13.2. **Strata project 0.1.43, commit `7695859a`, engine 0.1.43.**

## Method

`./bench.sh --config <arm> --url http://127.0.0.1:8090 --long` — three runs per test, medians with ranges, loading not
timed, one warm-up chat and 1K prompt discarded, every fresh prompt carries a unique tag so the conversation cache
cannot help it. **Each arm was run twice, 3–5 minutes apart**; the tables below are run 2. Before each pair I asserted
`/health` reports `loaded: true` and `/v1/models` returns the config's `model_name`. One parallel slot, thinking off,
temperature 0, `STRATA_PF_FUSED=1`, vision encoder resident on the 5060 Ti (no image was sent).

| arm | model / quant | mode | on disk | expert arena |
| --- | --- | --- | --- | --- |
| A   | `Qwen3.8-Flash-Next-GSQ-RCO-IQ3_XXS` | plain | 44 GB | 42,912,972,800 B (40.0 GiB) |
| B   | `Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S` | plain | 78 GB | 50,292,326,400 B (46.8 GiB) |
| C   | `unsloth/Qwen3.8-Flash-Next-GGUF` **UD-IQ4_XS**, rev `38bb39e`, 3 shards, all SHA-256 verified | mapped, `--resident-budget-gib 55` | 88 GB | 59,519,795,200 B (55.4 GiB) |

All three share `--expert-cache auto --prefill auto --layer-split auto --spec 4 --mtp rt --max-context 262144 --kv q4_0
--vision --vram-reserve-mib 700 --spec-min-p 0.5`. Arm C's pack was built with `tools/iq_pack.py --compat-bf16`
(1,079 tensors, 303 served natively, 460 conversions — 264 exact, 196 rounded, max |err| 0.0144; `native_experts.txt`
v4 with the per-role shard column). `tools/test_iq_pack.py` 24/24 OK. **Each arm ran at `layer_split auto`'s own
optimum — that is the comparison being made.**

## Results (medians; range over 3 runs; 9 requests for the chat arm)

| test | A · IQ3_XXS | B · IQ3_S | C · UD-IQ4_XS |
| --- | --- | --- | --- |
| 4K prompt, fresh (4,019 tok) | **1,295.9** (1259.6-1297.2) | 811.3 (810.8-811.5) | 519.8 (519.7-519.8) |
| 16K prompt, fresh (16,058 tok) | **2,689.9** (2684.6-2692.0) | 1,579.8 (1577.9-1580.7) | 925.5 (925.3-925.7) |
| 32K prompt, fresh (32,049 tok) | **3,227.8** (3224.6-3227.9) | 1,946.0 (1943.8-1948.9) | 1,269.5 (1268.9-1269.9) |
| TTFT at 32K | **9.96 s** | 16.50 s | 25.28 s |
| short chat decode (100 tok) | **111.3** (100.9-145.6) | 102.6 (94.1-139.2) | 90.9 (84.4-121.3) |
| short chat TTFT | 0.30 s | 0.32 s | 0.30 s |
| agent session decode | 109.8 (87.9-134.6) | **113.3** (92.6-132.7) | 96.1 (77.0-115.5) |
| agent follow-up turn TTFT | **2.61 s** | 4.45 s | 6.88 s |
| first agent turn (nothing cached) | **1.14 s** | 2.33 s | 3.27 s |
| agent cache hit | 68.7% | 68.8% | 68.8% |
| earlier-turn tokens re-read per session | 72  | 5   | 2   |
| suite total | 183 s | 193 s | 292 s |

Reproducibility, run 1 vs run 2 (32K prompt tok/s): A 3,227.8 / 3,227.8 · B 1,947.9 / 1,946.0 · C 1,268.3 / 1,269.5.
Prefill is reproducible to **0.1–0.3%** (arm A also: 4K 1,295.3 / 1,295.9, 16K 2,686.5 / 2,689.9); decode wanders ±8% run
to run on this box, which is why the decode column is read as a band, not a ranking. One run to discard: arm A's run 1
chat TTFT reads 64.96 s (range 0.30–83.83) because another client was queued on the server in front of that test — the
prompt-path numbers from that run are unaffected and are included above, its chat latency is not.

## What the engine logged (this is where the story is)

|     | A · IQ3_XXS | B · IQ3_S | C · UD-IQ4_XS |
| --- | --- | --- | --- |
| split `auto` chose | K=3,6,30 | K=3,7,29 | K=3,11,30 |
| measured decode — chat / agent tok/s | **111.3 / 109.8** | 102.6 / **113.3** | 90.9 / 96.1 |
| measured decode window — ms per token | **9.0 / 9.1** | 9.7 / 8.8 | 11.0 / 10.4 |
| `the caches hold N of 24576 profiled pairs` | **24,387** | **21,537** (99.7% of routed mass) | **20,132** |
| `layer split: NN% of the experts resident` | **98%** | **86%** | **81%** |
| `prompt chunk auto` | 4,096 | 6,144 | 7,424 |
| decode expert cache hit | 99.9–100.0% | 99.7–99.9% | 98.0–99.8% |
| RAM tier | none | none | 24.23 GiB page-locked in 16 s; **18,388 experts left on the mmap fallback** |

Decode came out within ±8% of itself across three arms whose expert arenas differ by 39% — **9.0 / 9.7 / 11.0 ms
per token** — while every other row in this table moved. All three arms held 98–100% decode cache hit, and arm C logged
`0 blob reads from the file` while generating.

## The finding

**Decode is flat across a 39% swing in expert bytes. Prefill is not, and it follows residency, not the quant label.**

- 98% → 86% → 81% resident, and 32K prefill 3,228 → 1,946 → 1,270 tok/s.
- **Super-linear:** losing 12 points of residency costs 40% of prefill; losing 17 points costs 61%. Once the streamed
  expert set stops fitting in VRAM on cards whose links probe 3.1 GB/s, each further point of spill costs roughly three
  times the one before it.
- **Not chunk clamping:** the prompt chunk moves the *wrong* way (4,096 → 6,144 → 7,424) while prefill falls.
- **Not decode thrash:** hit rates stay 98–100% and arm C never read a blob from the file during generation. The prompt
  path streams every expert once per chunk — that is the only path that pays.

In practical terms for a 76 GiB four-card box: a 100k-token agent turn with a 1k answer costs **~40 s (A) → ~65 s (B)
→ ~80 s (C)**. I use agent clients that resend a 90–130k conversation every turn, so I kept IQ3_XXS even though the
other two are "better" quants.

## For maintainers

**`docs/UNSLOTH_Q4.md` says the pack is not measured on NVIDIA yet.** These are those numbers: it loads, it is
  stable, decode is good, prefill is 2.5x off the same box's IQ3_XXS arm.

## Not measured

Correctness (speed suite only; no needle probes on any arm), prompts past 32K, `--clients` concurrency (one slot),
images (encoder resident, none sent), sampled decoding, llama.cpp on the same files, storage model, CPU temperature.
Arm C carries `--compat-bf16` rounding, so its quality is the pack's, not the GGUF's. Arm B's pack holds its own
`experts.bin` copy (46.8 GiB); arm A reads experts in place out of the GGUF — same NVMe, same page-cache path.

Per-arm reports (same hardware section, full per-request tables) are available as `bench/results/` folders in the
`2026-10-12-community-4x-rtx-5060-ti-qwen3-8-flash-next-{iq3-xxs,iq3-s,ud-iq4-xs}{,-2}/` shape `bench.sh` writes, with
the engine log excerpts the tool could not capture because every run used `--url`.