#6128 [Engine] 0.1.60 on Windows: with --vram-reserve-mib 700 the expert cache grows past the auto size and decode drops to 0.2-42 tok/s on an RTX 4090
open · @Pfannkuchensack · 3 comentários · No GitHub
BenchmarksSetup & installServer & APINVIDIA / CUDAModels & quantsWindowsLinux
Descrição
### Area [Engine] ### Desktop app, Terminal or source? Source (setup.py from a git checkout) ### What happened After the update from engine 0.1.50 to 0.1.60 (`windows-x86_64-cuda-cuda13`), the Coder on an RTX 4090 became very slow with the config setup wrote (`--vram-reserve-mib 700`). The engine ends up with more expert slots than its own auto estimate, and little or no VRAM is left. With `--vram-reserve-mib 1500` the same model runs at the old speed. Engine log lines at load, same config except the engine and the reserve: | Start | Engine | Reserve | `expert cache auto: ... ->` | final `expert cache` | `VRAM free with everything loaded` | decode tok/s | |---|---|---|---|---|---|---| | 2026-10-29 | 0.1.50 | 700 | 6327 slots (then "shrinking" twice) | 7500 slots, 14.27 GiB | 285 MiB | 129-162 (2026-10-27, same setup) | | 2026-11-17 02:13 | 0.1.50 | 700 | 6327 slots | 7192 slots, 13.69 GiB | 291 MiB | not measured | | 2026-11-17 02:33 | 0.1.60 | 700 | 6222 slots, then "only 484 MiB free ... shrinking" once | 7961 slots, 15.15 GiB | **0 MiB - LOW** | **0.2** (52 tokens in 423 s) | | 2026-11-17 03:15 | 0.1.60 | 700 | 6222 slots | 8106 slots, 15.43 GiB | 379 MiB | **19.2 / 41.5 / 25.9** | | 2026-11-17 02:47 | 0.1.60 | 1500 | 5907 slots | 7522 slots, 14.32 GiB | 1226 MiB | 124.7 / 131.1 / 136.9 / 138.6 | | 2026-11-17 03:03 | 0.1.60 | 1500 | 5907 slots | 7447 slots, 14.17 GiB | 1093 MiB | 97.0 / 98.0 / 110.8 | Two things I do not understand from the log: - On 0.1.60 the final cache (7961 and 8106 slots) is larger than the auto line's number (6222) and larger than on 0.1.50 (7192-7500), although 0.1.60 now subtracts "+256 MiB for what is allocated after the cache". - On 0.1.50 the card also ended with only ~290 MiB free and decoded fast; on 0.1.60, 379 MiB free still gave 19-42 tok/s. The first 0.1.60 start printed: `0 MiB of VRAM free with everything loaded - LOW: requests may stall; add --vram-reserve-mib 1212 to the config's args` and `WARNING: only 0 MiB of VRAM free at the start of this request (below 1024 MiB): when the card is this full the driver can page GPU memory to system RAM`. Expected: with the reserve setup writes, 0.1.60 keeps enough VRAM free to decode at the 0.1.50 speed, or setup writes a larger reserve for 0.1.60. ### Smallest steps or command that reproduce it 1. Source install, `--family coder`, RTX 4090, the config setup writes (`--vram-reserve-mib 700`, 262144 context, pictures on the GPU). 2. Engine 0.1.60, send the same prompt 3 times (600-token cap, `temperature: 0`, `reasoning_effort: "none"`). 3. Set `--vram-reserve-mib 1500` in the config's args, restart, repeat. ### Reproduction status Reproduced twice at 700 (two starts), and twice not seen at 1500. ### Strata version Checkout c77572806 (`strata 0.2.15 (build 0.1.61)`), engine 0.1.60 `windows-x86_64-cuda-cuda13` from the signed feed (sequence 127); 0.1.50 for the comparison rows. ### Graphics card, memory and system - RTX 4090 24 GB, driver 621.01 (WDDM). It drives the main monitor (2560x1440). - At the 02:33 start it drove two more monitors (1920x1080 each). At 03:15 those two were on the Ryzen iGPU. - An RX 9060 XT in the same PC (another Strata install, idle). - Ryzen 9 7900, 128 GB RAM, Windows 10 Pro 22H2 (19045). ### Model, size and settings Qwen3.8-Flash-Next Coder IQ1_M (2026-09-29 shard 1), `--max-context 262144 --kv int8 --kv-resident 32768`, `--expert-cache auto --prefill auto --spec 4 --spec-min-p 0.5`, MTP on, `--vision` (GPU encoder). ### Not tested The IQ3_S config (its reserve was raised to 1500 too, before any 0.1.60 run), contexts other than 262144, pictures off, the engines between 0.1.50 and 0.1.60, Linux. The 0.1.60 rows at 700 are only two starts, with the monitor setup different between them. Desktop programs (browser, VS Code, Discord) were running during all runs. Measured and written with an AI coding assistant; all numbers above are from runs and logs on this PC.
No site
Links install, modelos, releases.