#3278 [Benchmark] 2x RTX 3090 NVLink, Ryzen 9 3950X, 121 GiB: v0.2.0-beta with IQ3_S / UD-Q4_K_XL / UD-Q5_K_XL on the Q8_0 PLE table, int8 vs fp16 KV

Description

**Hardware:** 2x NVIDIA GeForce RTX 3090 24 GB (sm_86), NVLink bridge (NV4, 4 links x 14.06 GB/s), each card PCIe 4.0 x8, power limit 400 W. AMD Ryzen 9 3950X (16 cores / 32 threads, AVX2, no AVX-512). 121 GiB RAM. Model files and PLE table on a Samsung 970 PRO 1 TB NVMe. Ubuntu 26.04.1, kernel 7.0.0-31, driver 595.84. Source build in a CUDA 13.0 container (`-DCMAKE_CUDA_ARCHITECTURES=86`), engine run in Docker.

**Engine:** `v0.2.0-beta` (tag `3d1fb480`, engine 0.1.45.3) plus:
- #1937 (conversations to disk, head `6e71ac5d`, merged; 14 conflict hunks resolved).
- #1525 (`759c8b90`, applied cleanly).
- #3022 (STRATA_HC_Q8 trim).
- The decode half of #1099 (`STRATA_SPLIT_P2P=1`: the window hand-off goes over NVLink), ported by hand.
- A small fix of ours: `--ple-gguf` may name a shard that holds **only** the PLE table and belongs to another file set. The engine used to add it to the dense shards and then refused it with "additional shard must match the architecture-validated first shard's split metadata".

**Same answers as stock:** stock 0.2.0 (plus only the PLE fix) against the build above, on the deterministic gate (`--expert-cache 3000 --pcie-frac 0 --adapt-swaps 0 --prompt-cache 0 --no-prefill-borrow`; greedy, 3 short prompts plus 2 questions on a 20K-token prompt). The output was **identical on all three quants** (IQ3_S, UD-Q4_K_XL, UD-Q5_K_XL, each with the Q8_0 table). On UD-Q4_K_XL it was also identical with `STRATA_SM86_PERF=0`, with `STRATA_SPLIT_P2P=1`, with `STRATA_HANDOFF_PIPE=4`, and with `STRATA_AUX_CPUS=auto STRATA_STAGE_PIN=1`.

### What was run

Three quants, all on the **Q8_0 PLE table** from UD-Q5_K_XL shard 3 (sha256 `34efd79a…`, the same file as UD-Q6_K_XL's shard 3). The table is passed with `--ple-gguf` and read from the SSD (`--ple-io direct`). Every expert is held in RAM; nothing is mapped from the file.
- ISTA GSQ-RCO IQ3_S
- unsloth UD-Q4_K_XL
- unsloth UD-Q5_K_XL

**Flags:**

```
--layer-split 24 --trim-stage-weights --expert-cache auto --prefill auto --ple-io direct --ple-gguf <UD-Q5_K_XL-00003-of-00006.gguf>
--kv int8|fp16 --kv-resident 20480 (IQ3_S: 32768) --max-context 262144 --spec 4 --spec-min-p 0.7 (IQ3_S: 0.8) --mtp mtp/rt
--vision (vision on the CPU, 2048 tokens) --vram-reserve-mib 400 --pipeline-windows 2 --pcie-frac 0.2
--conversation-cache-mib 16384 / 32768 / 8192 (Q4 / IQ3_S / Q5) --conversation-cache-slots 4 / 8 / 4 --conversation-disk-dir <dir>
env: STRATA_ROUTE_TAIL_SKIP=0 STRATA_SPLIT_P2P=1 STRATA_HC_Q8=1   (the sm_86 switch table is on by default)
```

**Method:** one fresh engine per arm, arms run one after another (not interleaved).
- "Chat": 400-token answers to a code prompt, a prose prompt and a 50K-token chat. Median of 3 after 2 warm-up rounds.
- "FB": flashbench at 1K / 8K / 32K / 128K, greedy and sampled (temperature 1.0), 2 repeats.
- Prefill: fresh 1.5K / 4K / 20K real-code prompts, 2 each.
- Run-to-run noise on this box is about ±3–5 % on decode.

### Results (tokens/s)

| Setup | Chat decode code / prose / 50K | FB decode greedy 1K / 8K / 32K / 128K | FB prefill greedy 1K / 8K / 32K / 128K | 20K prefill | Expert hit | RAM free after load |
|---|---|---|---|---:|---:|---:|
| Reference: our old build, 0.1.43.1 + ports, UD-Q4_K_XL, IQ4_NL table, int8 | 109.3 / 97.7 / 98.2 | 127.3 / 136.4 / 127.5 / 121.6 | 373 / 1,649 / 2,624 / 3,143 | 2,359 | 95.4 % | 37 GiB |
| IQ3_S, int8 KV | 137.3 / 128.9 / 137.7 | 140.3 / 139.9 / 138.4 / 122.8 | 908 / 2,260 / 3,618 / 4,451 | 3,285 | 99.7 % | 62 GiB |
| IQ3_S, fp16 KV | 137.6 / 135.5 / 132.6 | 144.5 / 139.7 / 137.6 / 135.6 | 843 / 1,921 / 2,996 / 3,562 | 2,751 | 99.7 % | 59 GiB |
| UD-Q4_K_XL, int8 KV | 111.9 / 105.5 / 103.6 | 126.4 / 120.0 / 115.2 / 111.2 | 366 / 1,711 / 2,621 / 3,119 | 2,119–2,355 | 95.0 % | 36 GiB |
| UD-Q4_K_XL, fp16 KV | 115.1 / 107.4 / 105.2 | 122.2 / 121.4 / 119.2 / 112.7 | 356 / 1,526 / 2,266 / 2,647 | 1,906 | 94.3 % | 33 GiB |
| UD-Q4_K_XL, int8, its own IQ4_NL table (control) | 119.3 / 99.2 / 110.9 | 124.3 / 121.1 / 119.5 / 109.2 | 369 / 1,708 / 2,618 / 3,112 | 2,315 | 94.4 % | 37 GiB |
| UD-Q4_K_XL, int8, tail skip ON | 120.8 / 108.3 / 113.2 | 127.3 / 119.7 / 121.6 / 109.1 | 368 / 1,690 / 2,614 / 3,115 | 2,274 | 97.0 % | 37 GiB |
| UD-Q5_K_XL, int8 KV | 96.0 / 79.6 / 79.4 | 107.5 / 103.5 / 96.8 / 89.3 | 250 / 1,294 / 1,989 / 2,347 | 1,733 | 89.8 % | 16 GiB |
| UD-Q5_K_XL, int8, tail skip ON | 105.3 / 93.3 / 95.0 | 116.8 / 112.9 / 109.3 / 99.4 | 251 / 1,302 / 1,995 / 2,354 | 1,741 | 93.7 % | 16 GiB |
| UD-Q5_K_XL, fp16 KV | did not start: see below | | | | | |

A 10-question quiz (thinking on) scored 10/10 on every setup that started.

### Findings
- **The sm_86 default table on a 2-card layer split:** same answers as `STRATA_SM86_PERF=0` (gate above), and faster. UD-Q4_K_XL int8: chat decode 110.2 / 101.8 / 102.6 against 107.7 / 97.9 / 97.3 with the table off; FB greedy decode 122.2 / 115.0 / 113.9 / 114.4 against 107.5 / 109.1 / 102.4 / 103.2. That is the multi-GPU row the release notes did not have.
- **The Q8_0 table costs nothing measurable here** when it is read from the SSD: UD-Q4_K_XL with Q8_0 against its own IQ4_NL table is within noise on every column (PLE read probe 93K vs 149K rows/s).
- **UD-Q5_K_XL with fp16 KV and every expert in RAM does not fit 121 GiB.** The experts (91.6 GiB) loaded and the GPU cache filled, then systemd-oomd killed the desktop session under memory pressure and the engine exited. With int8 KV it runs and leaves about 16 GiB free.
- **The 0.2.0 build against our 0.1.43.1 build** (UD-Q4_K_XL, int8): chat decode is about 6 % faster and prompt speed is about equal. Flashbench decode is about 5–7 % slower (FB geomean 116–117 against 121–125, two runs of the old build).
- `STRATA_MTP_STAGE0=1`: the engine exits at start on this build (not investigated; it may be our #1099 port).
- `STRATA_MMQ_KQ=1`: changes the gate output, and the 20K prefill was slower here (2,187–2,225 against 2,317–2,355).
- `STRATA_HANDOFF_PIPE=4`: identical output, no measurable gain.
- `--kv-resident 32768` on UD-Q4_K_XL: the gate output changes, and the speed gain is small.
- **Route tail skip ON:** UD-Q5_K_XL decode +9–15 %, UD-Q4_K_XL +2–7 %, IQ3_S +1 % (99.7 % hit already).

**Not tested:** concurrency, needle/recall, KLD. Each arm was run once (no interleaved pairs), so read differences below about 5 % as noise.

*Everything in this report was done by Claude Opus 5.5 working as an agent on this machine: the merges, the builds, the measurements and this write-up.*