#6431 [Engine] 0.1.63 STRATA_L2_PF=1 default regresses NVFP4 batch decode by 8-12% on 2x RTX 5090 with PCIe-streamed experts

open · @MasterNobody · 4 Kommentare · Auf GitHub

BenchmarksServer & APIMulti-GPUAMD / HIPNVIDIA / CUDAModels & quants

Beschreibung

### Area

[Engine]  (speed, kernels, models, the server and API)

### Desktop app, Terminal or source?

Docker container (`ghcr.io/niko1221/strata:0.2.17-cuda13` - pulled on 2026-11-20 as `latest-cuda13`, which
resolved to 0.2.17: the runs record `engine_version 0.1.63` and commit `12d491da` in `results.json`; release
engine binary, `build_kind: release`).
Measured with the Terminal's standard benchmark against the running server (`strata bench run --url http://127.0.0.1:8080`, Strata 0.2.17, runner commit `12d491da`).

### What happened

Updating a production NVFP4 box from engine 0.1.62 (v0.2.16) to 0.1.63 (v0.2.17), decode throughput fell and the new
default-on `STRATA_L2_PF` of the sm_120 table is the cause on this configuration. Setting `STRATA_L2_PF=0` recovers
8-12% of decode and makes 0.1.63 equal-or-faster than 0.1.62 everywhere; the rest of the table is a net win here.

- Problem: on NVFP4 + `--batch 4` + 2x RTX 5090 with ~25% of experts streamed from disk (`pcie-frac 0.25`), the
  0.1.63 defaults decode 182.3 / 188.6 / 181.4 tok/s (short / 4k / 16k, medians) vs 208.1 / 198.2 / 185.8 on 0.1.62.
  This is a real regression, not run conditions: two independent cold-start baseline runs reproduce it to 0.3%.
- Bisect: `STRATA_L2_PF=0` restores decode to 200.5 / 209.9 / 196.7 and 196.2 / 200.8 / 204.2 in two independent
  runs. On the short case the two `L2_PF=0` ranges (min 190.7, 199.8) do not overlap the baseline ranges
  (max 185.5, 185.9) at all. `STRATA_POOL_DEP=0` is inert (medians within 0.2% of baseline).
  `STRATA_SM120_PERF=0` (the whole table off) is **worse** than the defaults (-6 to -12% decode): the rest of the
  sm_120 package (`EXP_MODE=8`, `Q80_ROWS1`, `HC_UPMIX`, and the pre-existing switches it also silences) is net
  positive on this box. The regression is specifically `L2_PF`.
- The release's L2-prefetch gain was measured on UD-Q4_K_XL on one RTX 5090 with DFlash2. With NVFP4 experts
  streamed over PCIe the decode window's waits look different and the side-stream prefetch appears to contend
  with them; that mechanism is **unconfirmed static analysis** (from reading `verify.cpp` and the gate at
  `STRATA_L2_PF`, [verify.cpp#L1509-L1522] in 0.2.17), not an observation.
- Quality is unaffected in every arm: 12/12 sanity checks; the switch is documented bitwise, this is speed only.

Complement to #6423: there, the same 0.1.63 package regressed prefill/TTFT on a mixed-arch layer split and
`SM120_PERF=0` recovered everything. Here the package is a net win and only one switch regresses, on a different
pack class and topology - the package's sign depends strongly on the workload profile, which argues for
per-switch gating rather than one table.

### Smallest steps or command that reproduce it

1. Serve the NVFP4 pack in the container - the full command is under "Model, size and settings" (API key
   removed); every arm is a fresh `docker run` of the identical command.
2. Wait for the model to load, then run `strata bench run --url http://127.0.0.1:8080` (required stage only).
   Decode medians come out ~182 / 188 / 181 tok/s.
3. Recreate the container adding `-e STRATA_L2_PF=0`, re-run the bench: decode comes back to ~196-210 tok/s
   across the three cases. (Recreating, not `docker restart`, so the environment actually changes.)

### Reproduction status

Reproduced locally. Two baseline runs, two `STRATA_L2_PF=0` runs (non-overlapping short-case ranges), one
`STRATA_POOL_DEP=0` run, one `STRATA_SM120_PERF=0` run. All seven runs on the same machine within 80 minutes on
2026-11-20 (baselines 15:49Z and 15:53Z; arms 16:33Z / 16:38Z / 16:41Z / 16:45Z), no other GPU load
(dedicated serving box), each run against a freshly recreated container.

### Strata version

Engine 0.1.63 release (`strata_commit 12d491da4a6904eb49aaa6f2f63ec4b1a17582e3`, `build_kind: release`).
Context arms on the same host and model: engine 0.1.62 (`...982627f8`, 2026-11-18T23:46Z) and engine 0.1.59
(`...77b45000`, 2026-11-18T23:25Z) runs of the same standard benchmark.

### Graphics card, memory and system

- 2x NVIDIA GeForce RTX 5090 32 GB (sm_120), both GPUs serving (the runner records GPU 0: 34,190,917,632 B,
  power limit 575 W); AMD Ryzen 9 9950X 16-Core; 121 GiB RAM; NVMe storage (drive identity is `null` in
  results.json; the PLE probe in the engine log reads random 4 KiB at ~652 MB/s).
- Host Ubuntu 24.04.5 LTS, kernel 7.0.0-42-generic, Docker; driver 595.99.02; CUDA 13.2. No power limits set
  by us; no other GPU workloads during the runs.

### Model, size and settings

- Model id `qwen3.8-flash-next-nvfp4-nvfp4`, quantization NVFP4, file `qwen-nvfp4-gptq-dense.gguf`
  (sha256 `abdf74336b53d8f7ca3d99947f1eac8f19b8a544966750127cb61e561b127e3b`). Model/source revisions: `null`
  (not pinned by the protocol draft).
- Context 262,144 tokens (`CONTEXT=256k`); `--parallel 4` (server_capacity 4); KV int8; vision on.
  Launch settings via `STRATA_SET` (become `strata launch --set` keys): `ple-io=direct`, `pcie-frac=0.25`,
  `conversation-cache-mib=14336`, `conversation-cache-slots=8`, `conversation-cache-min-tokens=8192`,
  `vram-reserve-mib=1024`, `vram-reserve-later-mib=1536`. Queue cap `max_queue 4`, `queue_timeout 120` in the
  run-config (idle at bench time). Engine env in every arm: `STRATA_BATCH_MTP=1`, `STRATA_PIPED_MTP=1`,
  `STRATA_BATCH_SELF_COMMIT=0`. Pack selected with `FAMILY=nvfp4 MODEL=NVFP4`, preset `nvfp4`.
- Per-arm additions (one env at a time, plain container environment): `STRATA_L2_PF=0` (x2),
  `STRATA_POOL_DEP=0`, `STRATA_SM120_PERF=0`.
- Sampling and prompts: the standard benchmark protocol `strata-qwen38-thinking-v1-draft.1`
  (profile `qwen3.8-flash-next-thinking`, thinking on, temp 1.0 / top_p 0.95 / top_k 20,
  reasoning effort xhigh, `corpus_sha256 e75eff8b...`).

```text
# container "strata-nvfp4"; no published ports - reachable only on the internal docker network
# "llm_shared_network" (fronted by LiteLLM); model path sanitized
docker run -d --name strata-nvfp4 --gpus all --restart unless-stopped \
  --network llm_shared_network --hostname strata \
  --ulimit memlock=-1:-1 --shm-size 2g --stop-timeout 120 \
  -v strata-data:/data \
  -v /srv/strata-models:/data/Strata-data \
  -e FAMILY=nvfp4 \
  -e MODEL=NVFP4 \
  -e CONTEXT=256k \
  -e VISION=on \
  -e PARALLEL=4 \
  -e STRATA_BATCH_MTP=1 \
  -e STRATA_PIPED_MTP=1 \
  -e STRATA_BATCH_SELF_COMMIT=0 \
  -e API_KEY=xxx \
  -e PRESET=nvfp4 \
  -e STRATA_SET="ple-io=direct conversation-cache-mib=14336 conversation-cache-slots=8 conversation-cache-min-tokens=8192 vram-reserve-mib=1024 vram-reserve-later-mib=1536 pcie-frac=0.25" \
  -e MALLOC_MMAP_THRESHOLD_=8388608 \
  ghcr.io/niko1221/strata:0.2.17-cuda13   # started from latest-cuda13, same content at the time
# each arm recreates this container with one added -e: STRATA_L2_PF=0 / STRATA_POOL_DEP=0 / STRATA_SM120_PERF=0
# (the last baseline-style run kept STRATA_L2_PF=0 in the line, hence it shows in docker inspect today)
```

### Measurements

Decode tok/s, per-case medians with the min-max of the 3 rounds inside each run. Every run: status `complete`,
0 failed requests, 12/12 sanity. Prompt-side metrics and TTFT are flat across all arms (prompt-4k
2,871-2,916 tok/s, prompt-16k 6,147-6,303, short TTFT 0.27-0.28 s, 4k TTFT 1.42 s, 16k TTFT 2.62-2.68 s;
`SM120_PERF=0` is the only arm 1.7% down at 16k prompt).

| run (UTC, arm) | short | prompt-4k | prompt-16k |
| --- | --- | --- | --- |
| 2026-11-18 23:25 · engine 0.1.59 | 184.2 (167.5-189.7) | 175.0 (160.9-180.1) | 174.1 (164.9-214.4) |
| 2026-11-18 23:46 · engine 0.1.62 | 208.1 (201.5-233.8) | 198.2 (193.1-211.6) | 185.8 (182.5-217.5) |
| 2026-11-20 15:49 · 0.1.63 defaults | 182.3 (178.1-185.9) | 188.6 (183.5-195.4) | 181.4 (161.5-201.4) |
| 2026-11-20 15:53 · 0.1.63 defaults | 182.1 (177.5-185.5) | 187.7 (183.4-195.4) | 181.7 (161.5-200.9) |
| 2026-11-20 16:33 · 0.1.63 + `L2_PF=0` | 200.5 (199.8-206.7) | 209.9 (176.2-215.3) | 196.7 (185.5-207.7) |
| 2026-11-20 16:45 · 0.1.63 + `L2_PF=0` | 196.2 (190.7-209.8) | 200.8 (185.2-207.9) | 204.2 (203.0-207.6) |
| 2026-11-20 16:38 · 0.1.63 + `POOL_DEP=0` | 182.4 (177.9-185.1) | 187.4 (183.6-195.4) | 181.4 (161.0-201.5) |
| 2026-11-20 16:41 · 0.1.63 + `SM120_PERF=0` | 171.6 (168.3-182.0) | 165.5 (157.4-184.3) | 173.7 (170.4-188.2) |

Against the two baseline runs: `L2_PF=0` is +8-10% short, +7-11% at 4k, +8-12% at 16k. The short case is the
clean one (no range overlap between arms); at 4k/16k individual rounds are noisier (every baseline run contains
one 16k round at ~161 tok/s) and the overlap is partial. `L2_PF=0` also matches or beats 0.1.62
(short within its range, 4k +1-6%, 16k +6-10%), so with the flag off the 0.1.63 update is a strict improvement
over 0.1.62 on this box. The 0.1.62 run is context only (it ran ~20 min after the 0.1.59 run, warm caches);
the `L2_PF` attribution rests entirely on the same-day arms.

The 0.1.63 15:53Z run also ran the optional concurrency stage (`--optional-concurrency`, c4 x3 rounds,
0 failed): aggregate 199 / 263 / 275 tok/s, median 76.0 tok/s per request, c4 TTFT 0.61 s. No concurrency arm
was run with `L2_PF=0`.

### Validation

- Passed: the six arms above (exact commands and revisions in the steps); 12/12 sanity checks in every run;
  required-stage duration 50.1-54.3 s per run; outputs truncated at the protocol's token budget are the
  protocol's normal outcome (3-4 `completed` of 9 performance requests per run), 0 failures.
- Failed: none observed.
- Not run: the concurrency stage with `L2_PF=0`; single-slot `--batch 1` (does the regression need batch
  slots?); `ROUTE_TAIL_SKIP=7` combined with `L2_PF` (tail skip is off here, as by default on NVFP4);
  `STRATA_L2_PF_MIB` / `_BLOCKS` tuning; any second host. Per-run `results.json` folders (seven) are available
  on request.

### Suggested fixes (for scope agreement)

- Gate `STRATA_L2_PF` off when it measured nothing comparable to this profile - e.g. when experts are streamed
  over PCIe (`pcie-frac > 0`) or the NVFP4 pack is in use, or with batch slots >= 2 - or, as a smaller step,
  document beside the release note that the prefetch's numbers are single-card all-resident and it can cost
  decode when experts stream.
- The start-up line it already prints (`STRATA_L2_PF=%d: ... L2 prefetch ...`) makes the state auditable, which
  helped this bisect.

### Limits

One host, one pack (NVFP4 dense GGUF), two repeats per key arm; individual rounds are noisy at 4k/16k and the
full ranges are kept above. The mechanism (prefetch contending with expert streaming) is unconfirmed static
analysis. No claim is made for other cards or packs: the direction may well be positive where the release
measured it.

Mehr auf der Site

Links zu Install, Modellen, Releases.