#2999 bench: community report, 4x Tesla T4 (sm_75, Linux, Docker): UD-Q6_K_XL on 0.1.45, +17% decode vs 0.1.42, 128K context verified with a 93K prompt, and the per-request VRAM warning that `--expert-cache auto` sets off by itself

本文

**Headline:** 0.1.45 decodes at **21.7 tok/s** on 4x T4 with UD-Q6_K_XL, compared with 18.6 on 0.1.42 (+17%). Prefill is unchanged at about 173 tok/s on a 1.7K prompt. At `--max-context 131072`, a **93,111-token prompt** was read at **220.8 tok/s** (433 s), then decoded at 18.6 tok/s, and the model answered a needle question correctly. The only rough edge: the "only ~448 MiB of VRAM free at the start of this request (below 512 MiB)" warning (#2065) fires on every request, even though the auto expert cache sized the cards itself. T4 sm_75 is now stable for us (the #1577 verify-window stall is gone).

## Hardware and software

- **GPUs:** 4x Tesla T4 16 GB (sm_75), PCIe Gen3 x16, power limit 70 W each, driver 615.71.09
- **CPU / RAM:** Xeon Silver 4314 (32C/64T), 251 GiB RAM
- Linux, rootless Docker, image built from the repo Dockerfile, CUDA 13.0 base
- **Strata v0.1.45, commit `477ac13c`.** Engine built from source for sm_75:
  `-DSTRATA_ENABLE_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=75 -DSTRATA_Q6K_EXPERTS=ON -DSTRATA_MMQ_KQUANTS=ON`. This build is bind-mounted as `/custom`, with `STRATA_SM75_PERF=1`.
- **Model:** `Qwen3.8-Flash-Next-UD-Q6_K_XL`, six shards (~169 GB), packed with `--pack`

## Configuration

```
strata --serve --pack /data/packs/ud-q6_k_xl
       --native .../Qwen3.8-Flash-Next-UD-Q6_K_XL-00001-of-00006.gguf
       --expert-profile data/expert-profile.bin --expert-cache auto
       --prefill 512 --spec 2 --spec-min-p 0.5 --mtp /data/mtp/rt
       --max-context 131072 --kv q4_0 --vram-reserve-later-mib 1208 --layer-split auto
```
One slot, temperature 0. All numbers below come from the server's own `/metrics` per-request fields (`prompt_ms`, `decode_tok_s`), not from counting streamed chunks, because spec/MTP makes chunk counting undercount.

## Results

| Engine | Context | Decode tok/s (300-tok output, median of 3) | Prefill tok/s (1.7K prompt, median of 3) | Expert cache |
|---|---|---|---|---|
| 0.1.42 | 2,048 | 18.6 | 175 | ~35.5 GiB |
| **0.1.45** | 2,048 | **21.7** (20.1 to 21.9) | 173.2 | 35,479 MiB |
| 0.1.45 | 131,072 | 19.9 | 165.2 | 32,683 MiB |
| 0.1.45, **93,111-token prompt** | 131,072 | 18.6 | **220.8** | 32,683 MiB |

- Draft acceptance on 300-token outputs at 2K: 114 to 131 of 153 to 168 offered (74 to 85%). The expert hit rate is about 0.96 on short prompts, 0.85 on the 1.7K prompt, and **0.60** on a separate 48K prompt.
- Caveat: the 0.1.42 baseline used a similar but not identical prompt set. The 0.1.45 rows all use the same script.
- Going from 2K to 128K costs about 2.8 GiB of expert cache and about 8% decode on short prompts.
- Long-prompt scaling on T4: 48K at 222 tok/s (216 s), 93K at 221 tok/s (433 s). Prefill stays flat as the prompt grows, so for interactive use the practical limit on this rig is time, not memory.

## The VRAM warning

At 2K with no reserve flag, both 0.1.42 and 0.1.45 print:
```
strata serve: CUDA3: 260 MiB of VRAM free with everything loaded - LOW ... add --vram-reserve-later-mib 1208
strata serve: WARNING: only 448 MiB of VRAM free at the start of this request (below 512 MiB) ... (#2065)
```
With `--vram-reserve-later-mib 1208` as suggested, the load-time message becomes `CUDA1/2: 1052 MiB, CUDA3: 768 MiB free with everything loaded`. **But the per-request warning still fires at 448 to 450 MiB on every request.** We have not seen paging or a hang: decode stays at 18 to 22 tok/s, including on the 93K prompt.

Questions:
1. Should `--expert-cache auto` keep the 512 MiB per-request margin by itself, so a default config doesn't warn against its own sizing?
2. On 16 GB T4s with `--layer-split auto`, is `--vram-reserve-mib 768` the recommended setting, or is there a better one?
3. At about 450 MiB free with no slowdown, is this just a precaution, or should we expect the #2065 hang under sustained load?

Happy to run test builds or `tools/strata_bench.py --long` on this box if useful.