#2127 Community Benchmark: Q3_XXS on multi-Nvidia

本文

# Field report: 4x NVIDIA (5060 Ti + 2x RTX 3090 + 3060), Flash-Next GSQ-RCO IQ3_XXS, 262K context, `layer_split auto`

Two full runs of the built-in benchmark (`tools/strata_bench.py --long`, 3 runs per test) against the same already-running
server, 14 minutes apart, engine **0.1.43**, project commit `7695859a`. Speed only — no correctness suite was run here.

## Hardware and software

- **GPUs:** RTX 5060 Ti 16 GB + RTX 3090 24 GB + RTX 3090 24 GB + RTX 3060 12 GB = 76 GiB VRAM, all four selected
- **PCIe:** only the 5060 Ti is on a wide link — the engine probes **13.9 GB/s** on it (x8); **both 3090s and the 3060
  probe 3.1 GB/s** (one 3090 on an Oculink/x4 riser, the 3060 in a x4-shaped slot). The engine prices an expert miss
  at **x6.4** on three of the four cards.
- **Power limits:** both 3090s capped at **225 W** (`nvidia-smi -pl`); during generation they sit at 221 W with SM clocks
  at 1695-1755 MHz against a 1935-1950 ceiling, so the cap is binding. 3060 capped at 125/170 W.
- **CPU / RAM:** Xeon W-1350 @ 3.30 GHz (6C/12T), **123.3 GiB** RAM, NVMe system disk. Linux, driver 595.91.07, CUDA 13.2.
- **Model:** ISTA-DASLab **GSQ-RCO IQ3_XXS** (3.00 bpw, 75.8 GB, two shards; shard 2 is the 28.8 GB IQ4_NL n-gram table,
  read directly — the engine's probe reports 58,877 rows/s). Vision encoder on the 5060 Ti.

## Configuration (unchanged from the install, plain mode)

```text
strata --serve --pack packs/iq3_xxs --native ...GSQ-RCO-IQ3_XXS-00001-of-00002.gguf
       --ple-gguf ...GSQ-RCO-IQ3_XXS-00002-of-00002.gguf --expert-profile data/expert-profile.bin
       --expert-cache auto --prefill auto --spec 4 --mtp mtp/rt --max-context 262144
       --kv q4_0 --vision --vram-reserve-mib 700 --spec-min-p 0.5 --layer-split auto
```

`STRATA_PF_FUSED=1`. No `--resident-experts`, no low-RAM mode, no `--batch` (one parallel slot). Thinking off for the
suite, temperature 0. Model loading is not timed.

**What `layer_split auto` chose** (engine's own lines, best of 24,319 placements):

| stage | card | SMs @ clock | ms/layer | PCIe probe | layers | expert cache |
| --- | --- | --- | --- | --- | --- | --- |
| CUDA0 | RTX 3060 | 28 @ 1.84 GHz | 1.41 | 3.1 GB/s | 0-2 | helper |
| CUDA1 | RTX 5060 Ti | 36 @ 2.62 GHz | 0.77 | 13.9 GB/s | 3-5 | 1,536 slots (2.19 GiB) |
| CUDA2 | RTX 3090 | 82 @ 1.70 GHz | 0.52 | 3.1 GB/s | 6-29 | 12,121 slots (17.85 GiB) |
| CUDA3 | RTX 3090 | 82 @ 1.75 GHz | 0.50 | 3.1 GB/s | 30-47 + head | 8,889 slots (16.67 GiB) |

`K=2,3,29` was refused for the #1616 reason (a stage under three layers clamps the prompt chunk of every card);
`K=3,6,30` chosen, caches hold **24,387 of 24,576** profiled expert pairs, engine reports **98% of the experts
resident** (24,082 on the device graph) and **3,687 MiB of VRAM free** with everything loaded. Decode expert-cache
hit rate across the whole session: **99.9-100.0%**.

The `WARNING: prompt chunk 4096 tokens, not 6144` line is expected on this layout (CUDA1, the 16 GB card with 3
layers and 1,536 slots, cannot lend a 6144-chunk's 1,462 slots). Before the auto split this same machine was clamped
to **1280**-token chunks by a 12 GB card owning a single layer, and prefill was ~800/1,630/1,820 at 4K/32K/128K —
`auto` is worth +55 / +84 / +84% prefill here.

## Results — run 2 (medians, range over 3 runs; 9 requests for the chat arm)

| test | prompt tokens | reused | generated | prompt tok/s | decode tok/s | TTFT (s) |
| --- | --- | --- | --- | --- | --- | --- |
| short chat | 46 (43-51) | 0   | 100 | —   | **111.3** (100.9-145.6) | **0.30** (0.28-0.33) |
| 4K prompt, fresh | 4,019 | 0   | 24 cap | **1,295.9** (1,259.6-1,297.2) | —   | 3.12 |
| 16K prompt, fresh | 16,058 | 0   | 24 cap | **2,689.9** (2,684.6-2,692.0) | —   | 6.00 |
| 32K prompt, fresh | 32,049 | 0   | 24 cap | **3,227.8** (3,224.6-3,227.9) | —   | 9.96 |
| agent session, follow-up turns | 14,492 at the last turn | see below | ≤64 | see below | **109.8** (87.9-134.6) | 2.61 (1.69-2.92) |

Agent conversation cache: **68.7% hit** (68.6-68.8), **72 earlier-turn tokens re-read per session** (1-73); the
13,635 new tokens of the appended files are read once. First turn (nothing cached) TTFT 1.14 s.

| turn | prompt tokens | reused | read | prompt tok/s | TTFT | decode |
| --- | --- | --- | --- | --- | --- | --- |
| 0   | 609 | 0   | 609 | 534.2 | 1.156 | 88.4 |
| 1   | 3,702 | 621 | 3,081 | 1,095.3 | 2.831 | 134.6 |
| 2   | 5,235 | 3,695 | 1,540 | 847.9 | 1.836 | 98.6 |
| 3   | 8,679 | 5,299 | 3,380 | 1,164.1 | 2.925 | 106.7 |
| 4   | 11,603 | 8,743 | 2,860 | 1,076.5 | 2.680 | 118.8 |
| 5   | 14,474 | 11,628 | 2,846 | 1,102.9 | 2.605 | 131.6 |

## Run 1 vs run 2 — reproducibility, and one measurement artifact

| metric | run 1 | run 2 | change |
| --- | --- | --- | --- |
| chat decode tok/s | 112.1 (102.5-143.9) | 111.3 (100.9-145.6) | -0.7% |
| chat first token (s) | 64.96 (0.30-83.83) | **0.30** (0.28-0.33) | see note |
| 4K prompt tok/s | 1,295.3 (1,206.8-1,295.8) | 1,295.9 (1,259.6-1,297.2) | +0.0% |
| 16K prompt tok/s | 2,686.5 | 2,689.9 | +0.1% |
| 32K prompt tok/s | 3,227.8 | 3,227.8 | 0.0% |
| agent cache hit % | 68.71 | 68.66 | -0.1% |
| agent turn TTFT (s) | 2.65 | 2.61 | -1.5% |
| agent decode tok/s | 114.0 | 109.8 | -3.7% |

**Run 1's chat first-token times are queueing, not the engine.** This box also serves an interactive coding agent on
the same port; that client's requests were in front of the suite during run 1's chat arm. Run 1's prompt arms —
measured while nothing else was talking to the engine — match run 2 within 0.1%, and run 2's chat TTFT is 0.30 s.
Please read 0.30 s as this machine's short-request TTFT. Both runs are kept so the artifact is visible.

## What this rig's shape means for the numbers

- Prefill **rises** with context (1,296 at 4K → 3,228 at 32K): the chunk is 4096 and the streamed ring amortises over
  longer prompts.
- Decode is **flat in context** (111 at chat, 110 across the agent session) — the GDN/QSA attention is cheap on this model.
- The agent arm is the number that matters for agent clients: resending a 14,474-token conversation costs ~2,850
  prompt tokens, i.e. ~2.6 s to first token per turn at this size.
- Three of four cards are on 3.1 GB/s links, so this rig rewards keeping experts resident (98% here) far more than it
  rewards a bigger expert cache on the small cards.

## Not measured

Correctness/recall (speed suite only; needle checks are run separately on this rig and pass 15/15), `--clients`
concurrency (the server runs one slot, no `--batch`), prompts past 32K in this suite (`--long` stops at 32K), storage
model, and CPU temperature. A UD-IQ4_XS run on the same rig with the same settings follows as a second report.