#6431 [Engine] 0.1.63 STRATA_L2_PF=1 default regresses NVFP4 batch decode by 8-12% on 2x RTX 5090 with PCIe-streamed experts
open · @MasterNobody · 4 Kommentare · Auf GitHub
BenchmarksServer & APIMulti-GPUAMD / HIPNVIDIA / CUDAModels & quants
Beschreibung
### Area [Engine] (speed, kernels, models, the server and API) ### Desktop app, Terminal or source? Docker container (`ghcr.io/niko1221/strata:0.2.17-cuda13` - pulled on 2026-11-20 as `latest-cuda13`, which resolved to 0.2.17: the runs record `engine_version 0.1.63` and commit `12d491da` in `results.json`; release engine binary, `build_kind: release`). Measured with the Terminal's standard benchmark against the running server (`strata bench run --url http://127.0.0.1:8080`, Strata 0.2.17, runner commit `12d491da`). ### What happened Updating a production NVFP4 box from engine 0.1.62 (v0.2.16) to 0.1.63 (v0.2.17), decode throughput fell and the new default-on `STRATA_L2_PF` of the sm_120 table is the cause on this configuration. Setting `STRATA_L2_PF=0` recovers 8-12% of decode and makes 0.1.63 equal-or-faster than 0.1.62 everywhere; the rest of the table is a net win here. - Problem: on NVFP4 + `--batch 4` + 2x RTX 5090 with ~25% of experts streamed from disk (`pcie-frac 0.25`), the 0.1.63 defaults decode 182.3 / 188.6 / 181.4 tok/s (short / 4k / 16k, medians) vs 208.1 / 198.2 / 185.8 on 0.1.62. This is a real regression, not run conditions: two independent cold-start baseline runs reproduce it to 0.3%. - Bisect: `STRATA_L2_PF=0` restores decode to 200.5 / 209.9 / 196.7 and 196.2 / 200.8 / 204.2 in two independent runs. On the short case the two `L2_PF=0` ranges (min 190.7, 199.8) do not overlap the baseline ranges (max 185.5, 185.9) at all. `STRATA_POOL_DEP=0` is inert (medians within 0.2% of baseline). `STRATA_SM120_PERF=0` (the whole table off) is **worse** than the defaults (-6 to -12% decode): the rest of the sm_120 package (`EXP_MODE=8`, `Q80_ROWS1`, `HC_UPMIX`, and the pre-existing switches it also silences) is net positive on this box. The regression is specifically `L2_PF`. - The release's L2-prefetch gain was measured on UD-Q4_K_XL on one RTX 5090 with DFlash2. With NVFP4 experts streamed over PCIe the decode window's waits look different and the side-stream prefetch appears to contend with them; that mechanism is **unconfirmed static analysis** (from reading `verify.cpp` and the gate at `STRATA_L2_PF`, [verify.cpp#L1509-L1522] in 0.2.17), not an observation. - Quality is unaffected in every arm: 12/12 sanity checks; the switch is documented bitwise, this is speed only. Complement to #6423: there, the same 0.1.63 package regressed prefill/TTFT on a mixed-arch layer split and `SM120_PERF=0` recovered everything. Here the package is a net win and only one switch regresses, on a different pack class and topology - the package's sign depends strongly on the workload profile, which argues for per-switch gating rather than one table. ### Smallest steps or command that reproduce it 1. Serve the NVFP4 pack in the container - the full command is under "Model, size and settings" (API key removed); every arm is a fresh `docker run` of the identical command. 2. Wait for the model to load, then run `strata bench run --url http://127.0.0.1:8080` (required stage only). Decode medians come out ~182 / 188 / 181 tok/s. 3. Recreate the container adding `-e STRATA_L2_PF=0`, re-run the bench: decode comes back to ~196-210 tok/s across the three cases. (Recreating, not `docker restart`, so the environment actually changes.) ### Reproduction status Reproduced locally. Two baseline runs, two `STRATA_L2_PF=0` runs (non-overlapping short-case ranges), one `STRATA_POOL_DEP=0` run, one `STRATA_SM120_PERF=0` run. All seven runs on the same machine within 80 minutes on 2026-11-20 (baselines 15:49Z and 15:53Z; arms 16:33Z / 16:38Z / 16:41Z / 16:45Z), no other GPU load (dedicated serving box), each run against a freshly recreated container. ### Strata version Engine 0.1.63 release (`strata_commit 12d491da4a6904eb49aaa6f2f63ec4b1a17582e3`, `build_kind: release`). Context arms on the same host and model: engine 0.1.62 (`...982627f8`, 2026-11-18T23:46Z) and engine 0.1.59 (`...77b45000`, 2026-11-18T23:25Z) runs of the same standard benchmark. ### Graphics card, memory and system - 2x NVIDIA GeForce RTX 5090 32 GB (sm_120), both GPUs serving (the runner records GPU 0: 34,190,917,632 B, power limit 575 W); AMD Ryzen 9 9950X 16-Core; 121 GiB RAM; NVMe storage (drive identity is `null` in results.json; the PLE probe in the engine log reads random 4 KiB at ~652 MB/s). - Host Ubuntu 24.04.5 LTS, kernel 7.0.0-42-generic, Docker; driver 595.99.02; CUDA 13.2. No power limits set by us; no other GPU workloads during the runs. ### Model, size and settings - Model id `qwen3.8-flash-next-nvfp4-nvfp4`, quantization NVFP4, file `qwen-nvfp4-gptq-dense.gguf` (sha256 `abdf74336b53d8f7ca3d99947f1eac8f19b8a544966750127cb61e561b127e3b`). Model/source revisions: `null` (not pinned by the protocol draft). - Context 262,144 tokens (`CONTEXT=256k`); `--parallel 4` (server_capacity 4); KV int8; vision on. Launch settings via `STRATA_SET` (become `strata launch --set` keys): `ple-io=direct`, `pcie-frac=0.25`, `conversation-cache-mib=14336`, `conversation-cache-slots=8`, `conversation-cache-min-tokens=8192`, `vram-reserve-mib=1024`, `vram-reserve-later-mib=1536`. Queue cap `max_queue 4`, `queue_timeout 120` in the run-config (idle at bench time). Engine env in every arm: `STRATA_BATCH_MTP=1`, `STRATA_PIPED_MTP=1`, `STRATA_BATCH_SELF_COMMIT=0`. Pack selected with `FAMILY=nvfp4 MODEL=NVFP4`, preset `nvfp4`. - Per-arm additions (one env at a time, plain container environment): `STRATA_L2_PF=0` (x2), `STRATA_POOL_DEP=0`, `STRATA_SM120_PERF=0`. - Sampling and prompts: the standard benchmark protocol `strata-qwen38-thinking-v1-draft.1` (profile `qwen3.8-flash-next-thinking`, thinking on, temp 1.0 / top_p 0.95 / top_k 20, reasoning effort xhigh, `corpus_sha256 e75eff8b...`). ```text # container "strata-nvfp4"; no published ports - reachable only on the internal docker network # "llm_shared_network" (fronted by LiteLLM); model path sanitized docker run -d --name strata-nvfp4 --gpus all --restart unless-stopped \ --network llm_shared_network --hostname strata \ --ulimit memlock=-1:-1 --shm-size 2g --stop-timeout 120 \ -v strata-data:/data \ -v /srv/strata-models:/data/Strata-data \ -e FAMILY=nvfp4 \ -e MODEL=NVFP4 \ -e CONTEXT=256k \ -e VISION=on \ -e PARALLEL=4 \ -e STRATA_BATCH_MTP=1 \ -e STRATA_PIPED_MTP=1 \ -e STRATA_BATCH_SELF_COMMIT=0 \ -e API_KEY=xxx \ -e PRESET=nvfp4 \ -e STRATA_SET="ple-io=direct conversation-cache-mib=14336 conversation-cache-slots=8 conversation-cache-min-tokens=8192 vram-reserve-mib=1024 vram-reserve-later-mib=1536 pcie-frac=0.25" \ -e MALLOC_MMAP_THRESHOLD_=8388608 \ ghcr.io/niko1221/strata:0.2.17-cuda13 # started from latest-cuda13, same content at the time # each arm recreates this container with one added -e: STRATA_L2_PF=0 / STRATA_POOL_DEP=0 / STRATA_SM120_PERF=0 # (the last baseline-style run kept STRATA_L2_PF=0 in the line, hence it shows in docker inspect today) ``` ### Measurements Decode tok/s, per-case medians with the min-max of the 3 rounds inside each run. Every run: status `complete`, 0 failed requests, 12/12 sanity. Prompt-side metrics and TTFT are flat across all arms (prompt-4k 2,871-2,916 tok/s, prompt-16k 6,147-6,303, short TTFT 0.27-0.28 s, 4k TTFT 1.42 s, 16k TTFT 2.62-2.68 s; `SM120_PERF=0` is the only arm 1.7% down at 16k prompt). | run (UTC, arm) | short | prompt-4k | prompt-16k | | --- | --- | --- | --- | | 2026-11-18 23:25 · engine 0.1.59 | 184.2 (167.5-189.7) | 175.0 (160.9-180.1) | 174.1 (164.9-214.4) | | 2026-11-18 23:46 · engine 0.1.62 | 208.1 (201.5-233.8) | 198.2 (193.1-211.6) | 185.8 (182.5-217.5) | | 2026-11-20 15:49 · 0.1.63 defaults | 182.3 (178.1-185.9) | 188.6 (183.5-195.4) | 181.4 (161.5-201.4) | | 2026-11-20 15:53 · 0.1.63 defaults | 182.1 (177.5-185.5) | 187.7 (183.4-195.4) | 181.7 (161.5-200.9) | | 2026-11-20 16:33 · 0.1.63 + `L2_PF=0` | 200.5 (199.8-206.7) | 209.9 (176.2-215.3) | 196.7 (185.5-207.7) | | 2026-11-20 16:45 · 0.1.63 + `L2_PF=0` | 196.2 (190.7-209.8) | 200.8 (185.2-207.9) | 204.2 (203.0-207.6) | | 2026-11-20 16:38 · 0.1.63 + `POOL_DEP=0` | 182.4 (177.9-185.1) | 187.4 (183.6-195.4) | 181.4 (161.0-201.5) | | 2026-11-20 16:41 · 0.1.63 + `SM120_PERF=0` | 171.6 (168.3-182.0) | 165.5 (157.4-184.3) | 173.7 (170.4-188.2) | Against the two baseline runs: `L2_PF=0` is +8-10% short, +7-11% at 4k, +8-12% at 16k. The short case is the clean one (no range overlap between arms); at 4k/16k individual rounds are noisier (every baseline run contains one 16k round at ~161 tok/s) and the overlap is partial. `L2_PF=0` also matches or beats 0.1.62 (short within its range, 4k +1-6%, 16k +6-10%), so with the flag off the 0.1.63 update is a strict improvement over 0.1.62 on this box. The 0.1.62 run is context only (it ran ~20 min after the 0.1.59 run, warm caches); the `L2_PF` attribution rests entirely on the same-day arms. The 0.1.63 15:53Z run also ran the optional concurrency stage (`--optional-concurrency`, c4 x3 rounds, 0 failed): aggregate 199 / 263 / 275 tok/s, median 76.0 tok/s per request, c4 TTFT 0.61 s. No concurrency arm was run with `L2_PF=0`. ### Validation - Passed: the six arms above (exact commands and revisions in the steps); 12/12 sanity checks in every run; required-stage duration 50.1-54.3 s per run; outputs truncated at the protocol's token budget are the protocol's normal outcome (3-4 `completed` of 9 performance requests per run), 0 failures. - Failed: none observed. - Not run: the concurrency stage with `L2_PF=0`; single-slot `--batch 1` (does the regression need batch slots?); `ROUTE_TAIL_SKIP=7` combined with `L2_PF` (tail skip is off here, as by default on NVFP4); `STRATA_L2_PF_MIB` / `_BLOCKS` tuning; any second host. Per-run `results.json` folders (seven) are available on request. ### Suggested fixes (for scope agreement) - Gate `STRATA_L2_PF` off when it measured nothing comparable to this profile - e.g. when experts are streamed over PCIe (`pcie-frac > 0`) or the NVFP4 pack is in use, or with batch slots >= 2 - or, as a smaller step, document beside the release note that the prefetch's numbers are single-card all-resident and it can cost decode when experts stream. - The start-up line it already prints (`STRATA_L2_PF=%d: ... L2 prefetch ...`) makes the state auditable, which helped this bisect. ### Limits One host, one pack (NVFP4 dense GGUF), two repeats per key arm; individual rounds are noisy at 4k/16k and the full ranges are kept above. The mechanism (prefetch contending with expert streaming) is unconfirmed static analysis. No claim is made for other cards or packs: the direction may well be positive where the release measured it.
Mehr auf der Site
Links zu Install, Modellen, Releases.