#1846 --expert-cache auto reports 0.00 GiB free (0 slots) on Windows/WDDM when the dGPU drives no display, while ~15 GiB is actually free

本文

### Summary
With `--expert-cache auto`, the engine sizes the VRAM expert tier to `0.00 GiB free`
(0 slots) and then dies with `the embedding buffer failed` — but the card actually has
7–15 GiB free (verified with nvidia-smi, cudaMemGetInfo and serve/telemetry in the same
run/environment). Forcing a fixed `--expert-cache N` starts fine, so the allocation path
is OK; only the `auto` free-VRAM sizing is wrong here.

### Environment
- OS: Windows 11 (WDDM)
- GPU: NVIDIA RTX 5070 Ti 16 GB (16303 MiB). **The display output was moved to the
  motherboard iGPU, so the dGPU drives no display.**
- RAM: 64 GB DDR5-6000
- Strata: 0.1.40 (engine log: `session is up (engine 0.1.40)`), CUDA build
- Model: Qwen3.8-Flash-Next GSQ-RCO IQ3_S (single GPU), pack `iq3_s`
- Engine args (sanitized):
  `--serve --pack <iq3_s> --native <shard1> --ple-gguf <shard2> --expert-profile expert-profile.bin
   --expert-cache auto --prefill auto --spec 4 --mtp <rt> --max-context 131072 --kv int8
   --vision --vram-reserve-mib 256 --pcie-frac 0.00 --spec-min-p 0.50 --pool-workers 15`

### What happens
```
strata generate: expert cache auto: 0.00 GiB free, 256 MiB reserved (+218 MiB for the draft head) -> 0 slots
strata generate: no VRAM is left for the expert cache: it needs at least 250 slots (634 MiB),
                 about 1110 MiB more than this card has free. To make room: a smaller --max-context
                 (the session, mostly its KV cache, takes 1931 MiB at 131072 tokens), --kv q4_0, ...
strata generate: session is up (engine 0.1.40)
strata generate: the embedding buffer failed
```
→ the process exits before ready.

### But the memory IS free (measured during the failing run)
- `nvidia-smi --query-gpu=memory.used,memory.free`, sampled every 6 s for the whole load:
  peak `used 8797 MiB / free 7199 MiB` (only `strata-vision.exe` + `strata.exe` on the card).
- A fresh CUDA context on the same box right after: `cudaMemGetInfo` →
  `free 15037 MiB / total 16275 MiB`.
- `serve.telemetry.free_vram_mib(0)` → `15995`.
- Idle: `0 MiB used, 15995 MiB free`.

So the engine's own auto-sizing number (0) disagrees with NVML and CUDA by ~7–15 GiB.

### Intermittent
Same config, same session: an earlier run logged
`expert cache auto: 6.58 GiB free ... -> 2467 slots` and reached `session is up`
(then hit `prefill gemm: cublasCreate: cuBLAS status 1`), while later runs logged
`0.00 GiB free`. Before moving the display to the iGPU, runs consistently reported
6.6–9.4 GiB free. The move to iGPU-primary seems to be the trigger.

### Workaround (confirmed)
Forcing a fixed size works:
```
--expert-cache 1500
→ [strata] filling the GPU's expert cache (1934 experts, 3.72 GiB of VRAM) ...
→ ready: http://127.0.0.1:8085/v1 (context 131072 tokens, images on)
→ strata serve: 2389 MiB of VRAM free with everything loaded
```
So only the `auto` free-VRAM detection is broken; a fixed `--expert-cache` allocates
correctly from the real free memory.

### Suspected area
The free-VRAM query that feeds `--expert-cache auto` (the sizing / `expert_cache_headroom()`
path) appears to return ~0 on WDDM when the dGPU is not driving a display. `serve/telemetry.py`'s
NVML query returns the correct value in the same environment, so it looks specific to the
engine's own CUDA/WDDM free-memory read (or the moment it is sampled).

### Related
#765 (VRAM budgeting order), #1841 (auto sizing / WDDM paging cliff).

---

### Attachments

**A. Full engine log of a failing run (`--expert-cache auto`, exits before ready):**

```
load_hparams: Qwen-VL models require at minimum 1024 image tokens to function correctly on grounding tasks
load_hparams: if you encounter problems with accuracy, try adding --image-min-tokens 1024
load_hparams: more info: https://github.com/ggml-org/llama.cpp/issues/16842

strata-vision: model files loaded in 12.4 s
strata-vision: warmed up at 1024 image tokens in 0.6 s
[strata] 2026-10-10 04:45:46 engine started: strata.exe --serve --pack <packs>/iq3_s --native <model>/...IQ3_S-00001-of-00002.gguf --ple-gguf <model>/...IQ3_S-00002-of-00002.gguf --expert-profile <strata>/data/expert-profile.bin --expert-cache auto --prefill auto --spec 4 --mtp <strata-data>/mtp/rt --max-context 131072 --kv int8 --vision --vram-reserve-mib 256 --pcie-frac 0.00 --spec-min-p 0.50 --pool-workers 15
strata generate: this CPU has no AVX-512: the expert kernels run on AVX-2 (multi-token for the i-quant gate/up rows)
strata generate: native pack: <packs>/iq3_s experts (largest blob 2.66 MB), token embedding IQ4_XS in mapped host memory (322 MiB, 0.8 s)
strata generate: 1466 MiB of weights loaded from <packs>/iq3_s in 0.8 s (302 canonical tensors skipped: served natively)
strata generate: 300 native projection matrices, 2253.41 MiB of weights, in 2.4 s
strata generate: the SSD is kept awake while rows are read: one page of the table after 100 ms without a read, until 60 s after the last request (STRATA_SSD_KEEPALIVE=0 turns it off)
strata generate: profile <strata>/data/expert-profile.bin: 24576 ranked pairs, built for 24576 slots
strata generate: PLE on, table 320001536 rows of <model>/...IQ3_S-00002-of-00002.gguf
strata mtp: draft layer loaded, 950 MiB of VRAM (experts 675, dense 111), files read in 0.20 s (3974 MiB/s)
strata generate: experimental native Q5_K head, 521472000 bytes, in 0.8 s
strata generate: GPU 0: NVIDIA GeForce RTX 5070 Ti, compute capability 12.0
strata generate: expert arena read unbuffered (0 of 16 probe reads from the file cache; 1.7 GiB available, 51.3 GiB of files)
strata generate: expert arena: locked 1302 MiB via working-set minimum + VirtualLock; cudaHostRegister of the whole arena FAILED (out of memory); 47 slices pinned (45 GiB); large pages refused for 50295996416 B (GetLargePageMinimum=2097152, VirtualAlloc error 1314); using 4 KB pages
strata generate: loaded 46.84 GiB at 2.32 GiB/s
strata generate: 15 pool workers on logical processors 1,10,11,12,13,22,23,2,3,4,5,6,7,8,9, the host thread on 0 (draining too) (--host-core first)
strata generate: expert cache auto: 0.00 GiB free, 256 MiB reserved (+218 MiB for the draft head) -> 0 slots
strata generate: no VRAM is left for the expert cache: it needs at least 250 slots (634 MiB), about 1110 MiB more than this card has free. To make room: a smaller --max-context (the session, mostly its KV cache, takes 1931 MiB at 131072 tokens), --kv q4_0, the English draft subset (setup --draft-vocab en; the draft head takes 218 MiB now), images on the CPU, or close other programs that use the GPU
strata generate: pre-filled 0 of 0 slots from the profile in 0.0 s (0 MB/s); slot 0 verified
strata generate: 15 expert-pool workers + the host thread
strata generate: session is up (engine 0.1.40)
strata generate: the embedding buffer failed
```

**B. Same config, only `--expert-cache 1500` instead of `auto` → starts fine:**

```
[strata] ... engine started: strata.exe --serve --pack <packs>/iq3_s --native <model>/...IQ3_S-00001-of-00002.gguf --ple-gguf <model>/...IQ3_S-00002-of-00002.gguf --expert-profile <strata>/data/expert-profile.bin --expert-cache 1500 --prefill auto --spec 4 --mtp <strata-data>/mtp/rt --max-context 131072 --kv int8 --vision --vram-reserve-mib 256 --pcie-frac 0.00 --spec-min-p 0.50 --pool-workers 15
[strata] filling the GPU's expert cache (1934 experts, 3.72 GiB of VRAM) ...
[strata] almost ready ...
[strata] starting 1 MCP server for the web app's chat: web
ready: http://127.0.0.1:8085/v1  (OpenAI: /v1/chat/completions, Anthropic: /v1/messages, context 131072 tokens, images on)
strata generate: session is up (engine 0.1.40)
strata generate: token graph hit path: 1934 resident experts, decided on the device
strata serve: prompt chunk auto: 7168 tokens, a 199-slot ring
strata serve: the prompt path borrows 1711 CUDA0 cache slots (3.28 GiB)
strata verify: window up to 6 tokens, 75.0 MiB of device buffers
strata mtp: draft head over 106299 tokens (212.9 MiB)
strata serve: 2389 MiB of VRAM free with everything loaded
```

**C. `nvidia-smi` sampled every 6 s during the failing load (only `strata-vision.exe` + `strata.exe` on the card):**

```
baseline:                used 0 MiB    free 15995 MiB
t=6s   (vision only):    used 1060 MiB free 14936 MiB
t=18s  (vision+engine):  used 8129 MiB free 7867 MiB
t=24-42s:                used 8797 MiB free 7199 MiB
(then the engine exits, used -> 0)
```

**D. Free VRAM as seen by other tools on the same box (idle):**

```
nvidia-smi (NVML):                     total 16303  used 0     free 15995 MiB
cudaMemGetInfo (fresh CUDA context):   total 16275  used 1238  free 15037 MiB
serve.telemetry.free_vram_mib(0):      15995
```