#5876 [Engine] KV reservation before expert placement starves the expert cache on small cards

open · @archadrian · 5 comentarios · En GitHub

Server & APINVIDIA / CUDAModels & quantsDocumentationWindows

Descripción

### Area
[Engine]

### Desktop app, Terminal or source?
Source (build)

### What happened
When serving a q36-family model on an 11 GB card, the engine reserves the full fixed KV footprint for `--max-context` **before** placing experts. With a large window this silently starves the expert cache:

- `--max-context 98304`: engine reserves ~8.4 GB for KV, then places only **18 of 256** experts per layer into VRAM. Decode at 24K context: 27.7 t/s.
- `--max-context 32768`: places **116 of 256** experts per layer. Decode at 24K: 29.8 t/s.
- Prefill ~380 t/s in both cases.

The decode difference at 24K is small (~7%) because decode at that context is attention-bound on this card — but the placement numbers are the point: choosing a large window on a small card costs 6x resident experts with no warning. Expected: place experts first (or reserve KV lazily / grow on demand), or document the interaction so fit guidance can state it. If the current order is intentional, a one-line note in the docs is enough for us to cite it.

### Smallest steps or command that reproduce it
1. Build for sm_75: `cmake -DCMAKE_CUDA_ARCHITECTURES=75 -DSTRATA_MMQ_KQUANTS=ON -DSTRATA_MMQ_TQ_OVERLAY=ON -DSTRATA_MMQ_TQP=ON`
2. Serve with `--max-context 98304 --kv q4_0 --kv-grow --expert-vram 6` (env `STRATA_Q8_PACKED=0`, GPU pinned by UUID via `CUDA_DEVICE_ORDER=PCI_BUS_ID`).
3. Read the placement line in the engine log (experts resident per layer); run a 24K-context decode ladder.
4. Repeat with `--max-context 32768`.

### Reproduction status
Reproduced locally

### Strata version
engine 0.1.58 (source build from the PR #5595 branch, sm_75)

### Graphics card, memory and system
RTX 2080 Ti 11 GB · 123 GB RAM · Ubuntu 24.04

### Model, size and settings
Qwen3.6-35B-A3B FTQ-M (TQ2_T experts, 10.71 GiB GGUF, 41 layers x 256 experts) · windows above, serve layer

### Logs
The engine log prints `--kv q4_0` as "not an option of this engine" on this sm_75 q36 build (KV fixed-width there), which is why the 98K window reserves ~8.4 GB up front. Expert placement counts (18/256 vs 116/256) come from the engine's own placement output in the same logs.

Context: we publish fit verdicts for cards like this (boxy.fit), and our current guidance is "size --max-context to the workload" because of this interaction. A placement-policy change would make the guidance unnecessary; docs are fine too. Happy to re-run any candidate change against our full 2080 Ti ladder — all rows are banked.

En el sitio

Enlaces a install, modelos, releases.