问题 / #3215
#3215 [Engine] Forced 1536 MiB VRAM reserve overwrites user config on 0.2.0-beta, dropping decode speed from 80 to 50 tok/s
open · @mesosomeso · 3 评论 · 去 GitHub 看
BenchmarksAMD / HIPNVIDIA / CUDAModels & quantsWindows
说明
Problem: On Strata 0.2.0-beta (Windows 11), the engine/launcher forcibly resets and overwrites `--vram-reserve-mib` to `1536` during model loading, ignoring user configuration (`0` or lower values). On a dedicated compute GPU (where the display is driven by the CPU iGPU), this unnecessary 1.5 GB VRAM reserve truncates the expert cache and drops decode throughput by ~37%. Environment: - Strata version: 0.2.0-beta (engine 0.1.45.3) - OS: Windows 11 - CPU: AMD Ryzen 7 7700 (Display connected to motherboard; iGPU handles all desktop rendering & DWM) - GPU: NVIDIA GeForce RTX 5070 12GB (Dedicated compute GPU, 0% display overhead) - RAM: 64 GB DDR5 - Model: `swift1.5-qwen3.8-flash-next-125b-IQ3_S` - Relevant settings: `--vram-reserve-mib 0`, Vision disabled Reproduce: 1. Configure `swift1.5-qwen3.8-flash-next-125b-IQ3_S` on Windows 11. 2. In the model config JSON (`strata-*.json`), explicitly set `--vram-reserve-mib 0` (or `256` / `512`) to maximize VRAM expert cache, and ensure vision is disabled. 3. Launch the model via `strata launch` or the Strata 0.2.0 app. 4. Check the startup log and the JSON configuration during/after loading. Expected: The explicit `--vram-reserve-mib 0` setting is respected and kept without being overwritten, allowing the entire 12 GB VRAM to be utilized for the expert cache (achieving ~80 tok/s as in 0.1.45). Actual: During model loading, the reserve is forcibly recalculated/overwritten to `--vram-reserve-mib 1536`. 1.5 GB of VRAM is permanently locked away, reducing expert cache capacity and dropping decode speed from ~80 tok/s down to ~50 tok/s. Disabling vision does not reclaim this memory. Evidence: - Strata 0.1.45 (full VRAM utilized for expert cache): ~80 tok/s decode - Strata 0.2.0-beta (1536 MiB forced reserve): ~50 tok/s decode (~37% regression) - Config JSON is modified back to 1536 upon launch. Reproduction status: reproduced locally
本站相关内容
相关页面的快捷入口。