#2125 Community Benchmark: IQ3_S on multi-GPU NVIDIA

Beschreibung

# Field report: IQ3_S on NVIDIA — 4x RTX (5060 Ti + 2x 3090 + 3060), 262K context, plain mode. Plus the three-quant ladder on one box

Third arm of a controlled comparison on the same machine, same suite, same day: **IQ3_XXS vs IQ3_S vs UD-IQ4_XS**, each
run by `tools/strata_bench.py --long` twice. Headline: **decode barely moves across the three quants (111 / 103 / 91
tok/s); prompt processing collapses (3,228 / 1,946 / 1,270 tok/s at 32K), and it tracks how many experts the engine
could make resident (98% / 86% / 81%), not the quant name.** On a rig with three 3.1 GB/s links, the quant choice is a
*prefill* decision. We keep IQ3_XXS. No correctness suite was run.

## Hardware and software

- **GPUs:** RTX 5060 Ti 16 GB + RTX 3090 24 GB + RTX 3090 24 GB + RTX 3060 12 GB = 76 GiB VRAM, all four selected
- **PCIe:** the 5060 Ti probes **13.9 GB/s** (x8); both 3090s and the 3060 probe **3.1 GB/s** (Oculink/x4 riser, x4-shaped
  slot). The engine prices an expert miss at **x6.35** on three of the four cards.
- **Power:** both 3090s capped at 225 W, the 3060 at 125 W (the cap binds during generation).
- **CPU / RAM / disk:** Xeon W-1350 @ 3.30 GHz (6C/12T), **123.3 GiB** RAM, NVMe (1.9 TB, 484 GB free).
- Linux 7.0.0-38-generic, driver 595.91.07, CUDA 13.2. **Strata project 0.1.43, commit `7695859a`, engine 0.1.43.**
- Model: `Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S` (ISTA quantizer), two shards, 78 GiB on disk.

## Configuration

```
strata --serve --pack packs/iq3_s
       --native .../Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00001-of-00002.gguf
       --ple-gguf .../Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00002-of-00002.gguf
       --expert-profile data/expert-profile.bin --expert-cache auto --prefill auto --spec 4
       --mtp mtp/rt --max-context 262144 --kv q4_0 --vision --vram-reserve-mib 700 --spec-min-p 0.5
       --layer-split auto
```

`STRATA_PF_FUSED=1`, one parallel slot, thinking off, temperature 0, loading not timed, vision encoder on the 5060 Ti
(no image sent). **Plain mode** — no `--resident-budget-gib`, no `--resident-experts` — matching the IQ3_XXS control.
This pack has its own `experts.bin` (50,292,326,400 B = 46.8 GiB); the IQ3_XXS pack reads experts in place out of the
GGUF. Same NVMe, same page-cache path either way.

## What the engine chose

|     | IQ3_XXS | **IQ3_S (this report)** | UD-IQ4_XS |
| --- | --- | --- | --- |
| expert bytes | 42,912,972,800 (40.0 GiB) | **50,292,326,400 (46.8 GiB)** | 59,519,795,200 (55.4 GiB) |
| split chosen by auto | K=3,6,30 | **K=3,7,29** | K=3,11,30 |
| measured decode — chat / agent tok/s | **111.3 / 109.8** | **102.6 / 113.3** | 90.9 / 96.1 |
| measured decode window — ms per token | 9.0 / 9.1 | **9.7 / 8.8** | 11.0 / 10.4 |
| expert pairs cached | 24,387 / 24,576 | **21,537 / 24,576** (99.7% of routed mass) | 20,132 / 24,576 |
| experts resident | **98%** | **86%** | 81% |
| prompt chunk | 4,096 | **6,144** | 7,424 |
| RAM tier | none | **none** | 24.23 GiB page-locked, 18,388 experts on the mmap fallback |
| decode cache hit | 99.9-100.0% | **99.7-99.9%** | 98.0-99.8%, 0 blob reads from file |

Like the other arms, `auto` refused a placement with a stage under 3 layers (#1616) and took K=3,7,29 for 28.8 ms per
decode window. Decode hit rates stayed at 99.7-99.9% throughout both suites, so this arm was not thrashing either.

## Results — run 2 (medians, range over 3 runs; 9 requests for the chat arm)

| test | prompt tokens | reused | generated | prompt tok/s | decode tok/s | TTFT (s) |
| --- | --- | --- | --- | --- | --- | --- |
| short chat | 46 (43-51) | 0   | 100 | —   | **102.6** (94.1-139.2) | **0.32** (0.29-0.35) |
| 4K prompt, fresh | 4,019 | 0   | 24 cap | **811.3** (810.8-811.5) | —   | 4.97 |
| 16K prompt, fresh | 16,058 | 0   | 24 cap | **1,579.8** (1577.9-1580.7) | —   | 10.19 |
| 32K prompt, fresh | 32,049 | 0   | 24 cap | **1,946.0** (1943.8-1948.9) | —   | 16.50 |
| agent session, follow-up turns | 14,479 at the last turn | see below | ≤ 64 | see below | **113.3** (92.6-132.7) | 4.45 (3.69-4.71) |

Agent cache hit **68.8%** (68.7-68.8), earlier-turn tokens re-read per session **5** (4-74), first turn (nothing cached)
TTFT **2.33 s**. Suite total 193 s.

## Run 1 vs run 2

| metric | run 1 | run 2 |
| --- | --- | --- |
| 4K prompt tok/s | 809.8 (786.3-811.2) | 811.3 |
| 16K prompt tok/s | 1,579.8 | 1,579.8 |
| 32K prompt tok/s | 1,947.9 | 1,946.0 |
| chat decode tok/s | 105.1 (86.6-140.0) | 102.6 (94.1-139.2) |
| chat first token (s) | 0.32 | 0.32 |
| agent decode tok/s | 104.0 | 113.3 |
| agent turn TTFT (s) | 4.55 | 4.45 |

Prefill reproducible to 0.1%; decode wanders ±8% run to run as on every other arm on this box.

## The ladder — three quants, one box, one suite, each at `layer_split auto`'s own optimum

|     | IQ3_XXS | IQ3_S | UD-IQ4_XS (mapped, 55 GiB) |
| --- | --- | --- | --- |
| 4K prompt tok/s | **1,295.9** | 811.3 | 519.8 |
| 16K prompt tok/s | **2,689.9** | 1,579.8 | 925.5 |
| 32K prompt tok/s | **3,227.8** | 1,946.0 | 1,269.5 |
| TTFT at 32K | **9.96 s** | 16.50 s | 25.28 s |
| chat decode tok/s | **111.3** | 102.6 | 90.9 |
| agent decode tok/s | 109.8 | **113.3** | 96.1 |
| agent turn TTFT | **2.61 s** | 4.45 s | 6.88 s |
| first agent turn TTFT | **1.14 s** | 2.33 s | 3.27 s |
| chat TTFT | 0.30 s | 0.32 s | 0.30 s |

**Decode is a wash: 111 → 103 → 91 tok/s across a 39% swing in expert bytes, and the agent-session decode is flat
within noise (109.8 / 113.3 / 96.1, ranges 88-135).** Prefill is where the money is: **−38/−41/−40%** for IQ3_S over
IQ3_XXS, **−60/−66/−61%** for UD-IQ4_XS. A 100k-token agent turn with a 1k answer: **~40 s (XXS) → ~65 s (S) → ~80 s
(Q4)**.

The ordering tracks **resident experts** (98 / 86 / 81%) and cached pairs (24,387 / 21,537 / 20,132), not the quant
label — and it is **super-linear**: losing 12 points of residency costs 40% of prefill, losing 17 points costs 61%.
Once the streamed expert set stops fitting, each further point of spill costs about three times as much as the one
before it. Meanwhile the prompt chunk goes the *wrong* way (4,096 → 6,144 → 7,424), so this is not chunk clamping.

**Ask:** `layer_split auto` optimises the decode window (the decode it produced for all three arms lands within ±8% of
itself) and appears blind to
prompt-path expert streaming. Would scoring placements by *resident pairs* — or by a prompt-path term — be worth it on
narrow-link multi-GPU boxes? We can re-run any arm at a forced `--layer-split` to test it.

## Not measured

Correctness (speed suite only), prompts past 32K, `--clients` concurrency (one slot), images, sampled decoding, llama.cpp
comparison. Storage model and CPU temperature not recorded. The IQ3_XXS and UD-IQ4_XS arms are reported separately
(`strata-field-report-iq3-xxs-4gpu-2026-10-12.md`, `strata-field-report-ud-iq4-xs-4gpu-2026-10-12.md`) with the same
hardware section.