问题 / #3215

#3215 [Engine] Forced 1536 MiB VRAM reserve overwrites user config on 0.2.0-beta, dropping decode speed from 80 to 50 tok/s

open · @mesosomeso · 3 评论 · 去 GitHub 看

BenchmarksAMD / HIPNVIDIA / CUDAModels & quantsWindows

说明

Problem:
On Strata 0.2.0-beta (Windows 11), the engine/launcher forcibly resets and overwrites `--vram-reserve-mib` to `1536` during model loading, ignoring user configuration (`0` or lower values). On a dedicated compute GPU (where the display is driven by the CPU iGPU), this unnecessary 1.5 GB VRAM reserve truncates the expert cache and drops decode throughput by ~37%.

Environment:
- Strata version: 0.2.0-beta (engine 0.1.45.3)
- OS: Windows 11
- CPU: AMD Ryzen 7 7700 (Display connected to motherboard; iGPU handles all desktop rendering & DWM)
- GPU: NVIDIA GeForce RTX 5070 12GB (Dedicated compute GPU, 0% display overhead)
- RAM: 64 GB DDR5
- Model: `swift1.5-qwen3.8-flash-next-125b-IQ3_S`
- Relevant settings: `--vram-reserve-mib 0`, Vision disabled

Reproduce:
1. Configure `swift1.5-qwen3.8-flash-next-125b-IQ3_S` on Windows 11.
2. In the model config JSON (`strata-*.json`), explicitly set `--vram-reserve-mib 0` (or `256` / `512`) to maximize VRAM expert cache, and ensure vision is disabled.
3. Launch the model via `strata launch` or the Strata 0.2.0 app.
4. Check the startup log and the JSON configuration during/after loading.

Expected:
The explicit `--vram-reserve-mib 0` setting is respected and kept without being overwritten, allowing the entire 12 GB VRAM to be utilized for the expert cache (achieving ~80 tok/s as in 0.1.45).

Actual:
During model loading, the reserve is forcibly recalculated/overwritten to `--vram-reserve-mib 1536`. 1.5 GB of VRAM is permanently locked away, reducing expert cache capacity and dropping decode speed from ~80 tok/s down to ~50 tok/s. Disabling vision does not reclaim this memory.

Evidence:
- Strata 0.1.45 (full VRAM utilized for expert cache): ~80 tok/s decode
- Strata 0.2.0-beta (1536 MiB forced reserve): ~50 tok/s decode (~37% regression)
- Config JSON is modified back to 1536 upon launch.

Reproduction status:
reproduced locally

本站相关内容

相关页面的快捷入口。