#3470 [Engine] CUDA layer split, 0.1.44.1: later-stage (CUDA1) free VRAM drops 2-6 MiB per long cold prompt and does not plateau

Descripción

## Setup
- Strata v0.1.44.1 (tag, built locally: CUDA 13.0 nvcc, `CMAKE_CUDA_ARCHITECTURES=120`, Release), Linux, driver 595.91.07
- 2x RTX 5060 Ti 16 GB (sm_120), 48 GB RAM
- Model: Qwen3.8-Flash-Next GSQ-RCO IQ3_S pack, `--layer-split auto` (K=23: CUDA0 layers 0-22, CUDA1 23-47 + head + MTP)
- Args: `--expert-cache auto --prefill auto --spec 4 --mtp <rt> --max-context 262144 --kv int8 --resident-experts
  --resident-budget-gib 31.58 --vision --vram-reserve-mib 700 --remote-expert-opt --pcie-frac 0.35 --spec-min-p 0.70`
  (plus a control vector, not applied on these requests). `CUDA_MODULE_LOADING` unset (EAGER forced by generate).
- Startup: `CUDA1: 203 MiB of VRAM free with everything loaded - LOW ... add --vram-reserve-later-mib 1265`

## Observation
Free VRAM on CUDA1 (`nvidia-smi -i 1 --query-gpu=memory.free`) after each request, fresh process, single client:

| step | request | CUDA1 free (MiB) |
|---|---|---|
| load | - | 204 |
| 1 | 114,972-token prompt, cold | 142 (first-use verify graphs: 12 captures logged, none after) |
| 2 | same, cached | 142 |
| 3-10 | 8 distinct cold prompts, 99-120K tokens | 136, 132, 128, 126, 122, 118, 116, 112 |
| 11-16 | 114,972-token prompt: 1 cold + 5 cached | 110 |
| 17-19 | **the same 113,445-token prompt, cold 3x** (a 13-token request in between drops the prompt cache) | **106, 104, 98** |

- Steps 17-19 repeat an identical prompt (same length, same chunking, same window sizes) and still lose 4, 2 and 6 MiB.
- Flat while idle (an earlier process held 92 MiB overnight).
- No `capturing the N-token window` lines after the first prompt, so the verify graphs are not being recaptured.
- An earlier 0.1.44.1 process without `--resident-budget-gib` showed the same trend: 189 MiB at load, 126 after the first long
  prompt, 92 after 118 mixed requests.
- CUDA0: 330 MiB at load, 272 after the first prompt, still 272 after step 19.

## Source read (0.1.44.1)
No allocator in `src/` grows with prompt length or position:
- verify graphs are keyed by window size (`verify.cpp:1881-1887`), one extra set for the doorbell path;
- prompt-path buffers live in borrowed cache slots (`prefill.cpp:1082-1219`);
- the MTP `pf_dev_` buffer is a high-water mark of at most one chunk (`mtp.cpp:1403`);
- KV is preallocated for `--max-context` (`generate.cpp:4446`);
- cuBLAS uses fixed 32 MiB workspaces (`gemm.cu:441`);
- there is no `cudaMallocAsync` or memory pool.

That leaves state inside the driver or cuBLAS, outside the source.

## Mitigation tried: `--vram-reserve-later-mib 1265`
Same build and args, plus `--vram-reserve-later-mib 1265` (the value the startup warning suggests), with `--resident-budget-gib`
raised from 31.58 to 32.14 so that the experts dropped from the GPU caches still get a RAM copy.

| | without | with reserve-later 1265 |
|---|---|---|
| layer split auto | K=23 (CUDA1 9.79 GiB free before its session carve) | K=24 (CUDA1 9.23 GiB) |
| CUDA0 expert cache | 3316 slots (8.91 GiB free) | 3200 slots (8.62 GiB free) |
| CUDA1 expert cache | 3517 slots (7.05 GiB) | 3374 slots (6.78 GiB) |
| experts on the GPUs (token graph hit path) | 8097 | 7787 (-310, -3.8 %) |
| RAM copy | 16479 experts, 31.58 GiB | 16789 experts, 32.14 GiB |
| LOW VRAM warning at startup | yes (CUDA1 203 MiB) | no |
| free VRAM at load, CUDA0 / CUDA1 | 329 / 203 MiB | 328 / 768 MiB |
| after the first 114,972-token prompt | 272 / 142 MiB | 270 / 706 MiB |
| prefill, 114,972 tokens cold | 93.1 s, 83.7 s | 92.0 s, 91.2 s, 80.8 s |
| decode after a cold prompt | 47.6, 41.7 tok/s (and 52.0, 44.8 uncapped) | 36.3, 41.8, 43.6 tok/s (median 41.8) |
| decode, same prompt cached | 45.0-47.3 tok/s | 49.1 tok/s |

- The first-use drop is the same (about 62 MiB), so the reserve adds about 565 MiB of runway on CUDA1. At the observed
  2-6 MiB per long cold prompt, that is on the order of 100+ long prompts more than before.
- Prefill and cached decode are unchanged.
- Decode after a cold prompt has a median of 41.8 tok/s, at the low end of the earlier cold runs (41.7-52.0). I don't
  read it as a measurable cost of the 310 fewer GPU experts.
- CUDA0 loses slots too, although the reserve only applies to later stages, because it changes the split search
  (`generate.cpp:3800-3812`, `stage_room`). The search sees 0.56 GiB less room on CUDA1 and moves one layer to CUDA0
  (K=23 -> 24). CUDA0's own expert cache, sized after the split, then has 0.29 GiB less free: -116 slots. CUDA1 loses
  only 143 slots (not the ~275 the reserve alone would cost) because it no longer holds layer 23.

So for us, `--vram-reserve-later-mib` is a workable mitigation. It delays the problem but does not stop the drop, and we
also restart the server on a schedule when free VRAM falls below 250 MiB.

## Questions
1. Is a slow per-prompt growth on later stages known? What does a full later card do here: stall, OOM, or a fallback?
2. Is there a way to trim it without a restart?
3. Is `--vram-reserve-later-mib` (placement only, `generate.cpp:3808`) the intended mitigation?