#6489 [Engine] 16K cold prefill regression on a single RTX 4090 D (sm_89, IQ3_S): 0.1.55 → 0.1.60/0.1.63 (−21 %), unrelated to --prefill, --expert-cache or the route tail skip
open · @Alex1298890 · 2 comentarios · En GitHub
Setup & installNVIDIA / CUDAModels & quantsSecurityWindows
Descripción
On one PC with one RTX 4090 D, the 16K cold-prompt throughput of `tools/strata_bench.py` drops from ~4 600–4 730 t/s on engines 0.1.45/0.1.55 to ~3 669–3 692 t/s on engines 0.1.60/0.1.63 (−21 %), with the same model, pack and args. The 4K prompt is unchanged and decode actually improved. Inside 0.1.63 the figure is pinned at 3 684–3 692 across every parameter arm we tried (`--prefill auto:16384` vs `auto:32768`, `--expert-cache auto` vs `6800`, `STRATA_ROUTE_TAIL_SKIP` off/3/7), so it looks like a deterministic engine-path change, not a tuning interaction. Related to #6118 and #6264 as far as I can tell from outside. ## Machine (identical for every run) - GPU: NVIDIA GeForce RTX 4090 D, 24 564 MiB, driver 617.42, PCIe gen4 x16, power limit 425 W - CPU: 13th Gen Intel Core i9-13900KF (24 cores / 24 threads), 127.8 GiB RAM - OS: Windows 11 (10.0.26300); engine packages: prebuilt Windows CUDA 13 (archs 75/86/89/120) - Model: Qwen3.8-Flash-Next GSQ-RCO **IQ3_S** (ISTA-DASLab), `--pack iq3_s`, MTP draft layers (`--mtp rt`), context 524 288 (yarn, scale 2), `--kv int8 --kv-resident 32768`, `--vision`, control-vector projection, expert-profile Engine args (identical across the 0.1.55 / 0.1.60 / 0.1.63 runs except where the arm table says otherwise): ```text --pack iq3_s --native Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00001-of-00002.gguf --ple-gguf Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00002-of-00002.gguf --expert-profile expert-profile.bin --expert-cache auto --mtp rt --max-context 524288 --rope-scaling yarn --rope-scale 2 --kv int8 --kv-resident 32768 --vision --control-vector-scaled Qwen3.8-Flash-Next-experimental-speed-projection.gguf:1.0 --control-vector-layer-range 4 44 --cvec-mode project --cvec-dir per-layer --prefill auto:16384 --spec 5 --spec-min-p 0.70 --pcie-frac 0.20 --pool-workers 15 --adapt-every 1 --adapt-swaps 80 --adapt-decay 0.97 --adapt-async 1 --conversation-cache-mib 16384 --conversation-cache-slots 4 --conversation-cache-min-free-mib 1024 --vram-reserve-mib 1536 ``` ## The regression All runs: `tools/strata_bench.py` (suite 1, 3 runs per test, back-to-back on the same day 2026-11-21), 16K prompt cold (`reused_tokens: 0`), engine's own `prompt_tps`. | Engine | expert cache fill | 16K prompt t/s (3 runs) | median | 4K t/s | chat decode t/s | |---|---|---|---|---|---| | 0.1.45 | 6 874 / 13.04 GiB | 4 744.4 / 4 718.2 / 4 727.8 | **4 728** | 2 052 | 159.4 | | 0.1.55 | 6 765 / 12.82 GiB | 4 691.3 / 4 602.7 / 4 593.8 | **4 603** | 2 023 | 153.5 | | 0.1.60 | 7 442 / 14.15 GiB | 3 722.9 / 3 665.6 / 3 668.5 | **3 669** | 2 024 | 157.2 | | 0.1.63 | 7 442 / 14.15 GiB | 3 734.6 / 3 675.4 / 3 687.8 | **3 688** | 2 089 | 175.7 | - The −21 % step lands between 0.1.55 and 0.1.60 (the 0.2.10–0.2.14 prompt-path rewrite window). - 4K prefill is flat across all four engines (2 023–2 089). - 16K TTFT went from 3.43–3.54 s (0.1.45/0.1.55) to 4.34–4.42 s (0.1.60/0.1.63). - Decode on 0.1.63 is the best of the four (chat median 175.7), so the regression is specific to the long-prompt read. ## What I ruled out inside 0.1.63 (same day, same machine, one variable per arm) | Arm | 16K t/s median | |---|---| | `--prefill auto:16384`, `--expert-cache auto`, `STRATA_ROUTE_TAIL_SKIP=3` | 3 687.8 | | `--prefill auto:32768`, cache auto, skip=3 | 3 687.7 | | `--prefill auto:32768`, cache auto, skip off | 3 688.6 | | `--prefill auto:16384`, cache auto, skip=7 | 3 684.1 | | `--prefill auto:16384`, `--expert-cache 6800` (old-engine fill), skip=3 | 3 692.1 | The 16K figure stays within 3 684–3 692 (≈0.2 %) across all five arms — deterministic, and independent of the prefill chunk limit, the expert-cache size and the route tail skip. The tail skip only moves decode (chat median 123.0 off / 149.4 rank-7 / 167.7–175.7 rank-3), as expected. ## How to reproduce 1. One RTX 4090 D (sm_89), Windows 11, driver 617.42, 128 GB RAM. 2. IQ3_S pack of Qwen3.8-Flash-Next GSQ-RCO with the args above. 3. Install Strata 0.2.9 (engine 0.1.55) in one folder and 0.2.15/0.2.17 (engine 0.1.60/0.1.63) in another, same model files; start each with its own `run-<model>.bat`. 4. Run `tools/strata_bench.py` in each; compare the 16K `prompt_tps`. ## Questions / guesses - Is this the same root cause as #6118 (cold 50K −16.5 % on a 3060) and #6264 (gfx1030 MoE GEMM gate/up 3.4× slower after the in-house kernels)? The determinism (3 684–3 692 across arms) and the 4K-vs-16K split make me suspect a chunking or staging decision that only engages above some prompt length (~4K ok, 16K slow) on sm_89 single-card. - #6404 shows 0.1.60→0.1.63 prefill moving a lot on gfx1030; on sm_89 the two are identical (3 669 vs 3 688). - Happy to run any instrumented build or env matrix (e.g. `STRATA_PREFILL_*` switches, `--prefill` fixed sizes, per-chunk logging) on this box to bisect 0.1.55→0.1.60. Context: this is my daily driver; 0.1.63 decode/TTFT are otherwise the best I have measured here, so this is the one number keeping me honest about the upgrade.
En el sitio
Enlaces a install, modelos, releases.