#2408 [Bug]: CUDA OOM while restoring a long conversation-cache snapshot on a 2-GPU layer split

本文

### What happened

On a Windows host with an NVIDIA GeForce RTX 3080 Ti + Tesla V100-PCIE-32GB, Strata 0.1.40 failed while serving a conversation that had been restored from the conversation cache. The relevant run used a two-GPU layer split, `--layer-split 24`, `--conversation-cache-slots 2`, `--conversation-cache-mib 4096`, `--max-context 262144`, `--kv int8`, `--kv-resident 32768`, `--vram-reserve-mib 700`, and speculative decoding.

The engine log shows the cache restore immediately before the failure:

```text
strata serve: conversation cache: parked 148167 tokens in 1573.2 ms; parked=2 bytes=3335695744 evictions=8 snapshot_bytes=3097849640
strata serve: conversation cache: restored 53 tokens (checkpoint) in 70.2 ms; parked=2 bytes=3336615992
...
strata serve: conversation cache: parked 58 tokens in 77.9 ms; parked=2 bytes=475631160 evictions=8 snapshot_bytes=237785056 reused_kv_bytes=793728
strata serve: conversation cache: restored 148167 tokens (live) in 524.1 ms; parked=2 bytes=2737274704
strata serve: prompt 148295 tokens = 148167 reused + 128 read in 3229 ms (39.6 tok/s), 88 generated in 11781 ms (7.5 tok/s), drafts accepted 54 of 61, 6 checkpoints
CUDA error: out of memory
  current device: 1, in function ggml_cuda_set_device
  cudaSetDevice(physical_device)
```

The process did not return a normal response after this sequence and the runtime started a new engine process afterward.

### Expected behavior

A conversation-cache hit is an optimization. If restoring its running state would not leave the configured VRAM reserve on every device in the layer split, Strata should return the parked snapshot to the cache and reread the prompt, rather than turning the request into a process-fatal CUDA OOM.

### Why this appears restore-specific

- The failure follows `restored 148167 tokens (live)` in the same server log.
- The failing device is device 1, which is the later stage of the two-GPU split.
- The configured VRAM reserve is 700 MiB, but the restore path does not appear to gate admission using a free-VRAM check for every split device.
- This is distinct from a prompt-cache performance issue: the requested behavior is safe fallback, not guaranteed cache reuse.

### Reproduction details

This was observed on the hardware and configuration above while exercising a roughly 148K-token conversation with two conversation-cache slots. I do not yet have a minimal deterministic reproducer; the attached sequence is from the engine log. The failure is not claimed to occur on every request.

### Suggested direction

Before restoring a parked conversation, sample free VRAM on CUDA0 and each layer-split stage. If any sampled device is below `--vram-reserve-mib`, skip the restore, put the snapshot back into the conversation cache, and reread the prompt. If a VRAM sample itself fails, preserve the existing behavior rather than rejecting the cache hit based on an incomplete sample.