#5876 [Engine] KV reservation before expert placement starves the expert cache on small cards
open · @archadrian · 5 commentaires · Sur GitHub
Server & APINVIDIA / CUDAModels & quantsDocumentationWindows
Description
### Area [Engine] ### Desktop app, Terminal or source? Source (build) ### What happened When serving a q36-family model on an 11 GB card, the engine reserves the full fixed KV footprint for `--max-context` **before** placing experts. With a large window this silently starves the expert cache: - `--max-context 98304`: engine reserves ~8.4 GB for KV, then places only **18 of 256** experts per layer into VRAM. Decode at 24K context: 27.7 t/s. - `--max-context 32768`: places **116 of 256** experts per layer. Decode at 24K: 29.8 t/s. - Prefill ~380 t/s in both cases. The decode difference at 24K is small (~7%) because decode at that context is attention-bound on this card — but the placement numbers are the point: choosing a large window on a small card costs 6x resident experts with no warning. Expected: place experts first (or reserve KV lazily / grow on demand), or document the interaction so fit guidance can state it. If the current order is intentional, a one-line note in the docs is enough for us to cite it. ### Smallest steps or command that reproduce it 1. Build for sm_75: `cmake -DCMAKE_CUDA_ARCHITECTURES=75 -DSTRATA_MMQ_KQUANTS=ON -DSTRATA_MMQ_TQ_OVERLAY=ON -DSTRATA_MMQ_TQP=ON` 2. Serve with `--max-context 98304 --kv q4_0 --kv-grow --expert-vram 6` (env `STRATA_Q8_PACKED=0`, GPU pinned by UUID via `CUDA_DEVICE_ORDER=PCI_BUS_ID`). 3. Read the placement line in the engine log (experts resident per layer); run a 24K-context decode ladder. 4. Repeat with `--max-context 32768`. ### Reproduction status Reproduced locally ### Strata version engine 0.1.58 (source build from the PR #5595 branch, sm_75) ### Graphics card, memory and system RTX 2080 Ti 11 GB · 123 GB RAM · Ubuntu 24.04 ### Model, size and settings Qwen3.6-35B-A3B FTQ-M (TQ2_T experts, 10.71 GiB GGUF, 41 layers x 256 experts) · windows above, serve layer ### Logs The engine log prints `--kv q4_0` as "not an option of this engine" on this sm_75 q36 build (KV fixed-width there), which is why the 98K window reserves ~8.4 GB up front. Expert placement counts (18/256 vs 116/256) come from the engine's own placement output in the same logs. Context: we publish fit verdicts for cards like this (boxy.fit), and our current guidance is "size --max-context to the workload" because of this interaction. A placement-policy change would make the guidance unnecessary; docs are fine too. Happy to re-run any candidate change against our full 2080 Ti ladder — all rows are banked.
Sur le site
Liens install, modèles, releases.