#2129 Community Report: Q4_XS on Multi-GPU NVIDIA

本文

# Field report: UD-IQ4_XS on NVIDIA (first NVIDIA run, I think) — 4x RTX (5060 Ti + 2x 3090 + 3060), 262K context, mapped mode, 55 GiB RAM budget

**Headline: it runs, it is stable, decode is fine (91-96 tok/s), and prompt processing is 2.4-2.9x slower than the same
rig's IQ3_XXS arm (520 / 926 / 1,270 tok/s at 4K / 16K / 32K vs 1,296 / 2,690 / 3,228).** For an agent client that
resends a 90-130k-token conversation every turn, that is the whole decision and we are keeping IQ3_XXS. Two full
`tools/strata_bench.py --long` runs, 3 runs per test, medians below; run 1 and run 2 agree within 0.3% on every prompt
arm. No correctness suite was run.

## Hardware and software

- **GPUs:** RTX 5060 Ti 16 GB + RTX 3090 24 GB + RTX 3090 24 GB + RTX 3060 12 GB = 76 GiB VRAM, all four selected
- **PCIe:** the 5060 Ti probes **13.9 GB/s** (x8); both 3090s and the 3060 probe **3.1 GB/s** (an Oculink/x4 riser and a
  x4-shaped slot). The engine prices an expert miss at **x6.35** on three of the four cards.
- **Power:** both 3090s capped at 225 W, the 3060 at 125 W (the cap binds during generation).
- **CPU / RAM / disk:** Xeon W-1350 @ 3.30 GHz (6C/12T), **123.3 GiB** RAM, NVMe.
- Linux, driver 595.91.07, CUDA 13.2. **Strata project 0.1.43, commit `7695859a`, engine 0.1.43.**
- Model: `unsloth/Qwen3.8-Flash-Next-GGUF` **UD-IQ4_XS**, revision `38bb39e`, three shards 93.7 GB, all three SHA-256
  match the table in `docs/UNSLOTH_Q4.md`.

## Configuration

```
strata --serve --pack packs/ud-iq4_xs
       --native .../Qwen3.8-Flash-Next-UD-IQ4_XS-00001-of-00003.gguf
       --resident-budget-gib 55 --expert-profile data/expert-profile.bin --expert-cache auto
       --prefill auto --spec 4 --mtp mtp/rt --max-context 262144 --kv q4_0 --vision
       --vram-reserve-mib 700 --spec-min-p 0.5 --layer-split auto
```

`STRATA_PF_FUSED=1`, one parallel slot, thinking off, temperature 0, loading not timed. No `--ple-gguf`: shard 1 holds
0 tensors and shard 2 holds `per_layer_token_embd.weight` (IQ4_NL, [160, 320001536]), which the engine found by name,
as documented. Vision encoder on the 5060 Ti; **no image was sent** (the doc says images are untested on this pack).

Pack built with `tools/iq_pack.py --compat-bf16`: 1,079 tensors, **303 served natively**, arena 1.38 GiB, 460
conversions (**264 exact, 196 rounded, max |err| 0.0144**), `native_experts.txt` **v4** with the per-role
`gate,up,down` shard column on 14 rows, expert arena **59,519,795,200 bytes (55.4 GiB)**. `tools/test_iq_pack.py`
24/24 OK. The `--compat-bf16` rounding is on, so the 6-9% PPL penalty the doc notes on Strix Halo applies to this arm too.

## What the engine chose (this is where the story is)

|     | this pack | our IQ3_XXS arm, same four cards |
| --- | --- | --- |
| split chosen by auto | **K=3,11,30** (5060 Ti takes layers 3-10, 3,715 slots) | K=3,6,30 (5060 Ti takes 3 layers) |
| measured decode — chat / agent tok/s | **90.9 / 96.1** (11.0 / 10.4 ms per token) | 111.3 / 109.8 (9.0 / 9.1 ms) |
| expert pairs cached | **20,132 of 24,576** (~99.2% of the routed mass) | **24,387 of 24,576** |
| experts resident | **81%** | **98%** |
| RAM tier | 24.23 GiB page-locked + mapped in 16 s; **18,388 experts left on the mmap fallback** | none (plain mode) |
| prompt chunk | **7,424 tokens**, 96-slot ring, 5,566 cache slots borrowed | 4,096 tokens |
| decode cache hit | 98.0-99.8%, **0 blob reads from the file** | 99.9-100.0% |

So decode was in good shape: 81% resident, ~99% hit rate, nothing read from the SSD while generating. The prompt path
is the one that suffers, and it is not a chunk clamp (this arm got a *bigger* chunk than IQ3_XXS). Reading a prompt
streams every expert once per chunk, and 19% of them are not in VRAM on three cards whose links probe 3.1 GB/s.

## Results — run 2 (medians, range over 3 runs; 9 requests for the chat arm)

| test | prompt tokens | reused | generated | prompt tok/s | decode tok/s | TTFT (s) |
| --- | --- | --- | --- | --- | --- | --- |
| short chat | 46 (43-51) | 0   | 100 | —   | **90.9** (84.4-121.3) | **0.30** (0.28-0.34) |
| 4K prompt, fresh | 4,019 | 0   | 24 cap | **519.8** (519.7-519.8) | —   | 7.75 |
| 16K prompt, fresh | 16,058 | 0   | 24 cap | **925.5** (925.3-925.7) | —   | 17.38 |
| 32K prompt, fresh | 32,049 | 0   | 24 cap | **1,269.5** (1,268.9-1,269.9) | —   | 25.28 |
| agent session, follow-up turns | 14,494 at the last turn | see below | ≤ 64 | see below | **96.1** (77.0-115.5) | 6.88 (5.52-7.32) |

Agent cache hit **68.8%** (68.8-68.9), earlier-turn tokens re-read per session **2** (2-4), first turn (nothing cached)
TTFT 3.27 s. Suite total 292 s.

## Run 1 vs run 2

| metric | run 1 | run 2 |
| --- | --- | --- |
| 4K prompt tok/s | 518.1 | 519.8 |
| 16K prompt tok/s | 923.9 | 925.5 |
| 32K prompt tok/s | 1,268.3 | 1,269.5 |
| chat decode tok/s | 87.3 (61.8-129.6) | 90.9 (84.4-121.3) |
| chat first token (s) | 0.35 | 0.30 |
| agent decode tok/s | 88.1 | 96.1 |
| agent turn TTFT (s) | 6.88 | 6.88 |

Prefill is reproducible to 0.3%. Decode wanders ±8% run to run, same as our other arms on this box.

## Head to head with the same rig's IQ3_XXS arm (same suite, same day, both at `layer_split auto`'s own optimum)

|     | IQ3_XXS (plain) | UD-IQ4_XS (mapped, 55 GiB) | delta |
| --- | --- | --- | --- |
| 4K prompt tok/s | 1,295.9 | 519.8 | **-60%** |
| 16K prompt tok/s | 2,689.9 | 925.5 | **-66%** |
| 32K prompt tok/s | 3,227.8 | 1,269.5 | **-61%** |
| chat decode tok/s | 111.3 | 90.9 | -18% |
| agent decode tok/s | 109.8 | 96.1 | -12% |
| TTFT at 32K | 9.96 s | 25.28 s | **+154%** |
| agent turn TTFT | 2.61 s | 6.88 s | +164% |
| chat TTFT | 0.30 s | 0.30 s | 0   |

A 100k-token agent turn with a 1k-token answer goes from **~40 s to ~80 s**. The +39% of expert bytes
(42.9 → 55.4 GiB) does not fit in 76 GiB of VRAM next to a 262K q4_0 KV cache, and on this box a miss costs 6.4x
what the engine's default `pcie_frac` assumes.

## Not measured

Correctness (speed suite only; needle probes not run on this pack), prompts past 32K, `--clients` concurrency (one
slot), images, sampled decoding, and any comparison against llama.cpp on the same file. `--compat-bf16` rounding is on,
so quality is the pack's, not the GGUF's. Storage model and CPU temperature not recorded.