Pull requests / #2409

#2409 fix: avoid CUDA OOM during conversation-cache restore

open · @MarsLuay · 0 Kommentare · Auf GitHub

BenchmarksServer & APIMulti-GPUNVIDIA / CUDA

Beschreibung

Fixes #2408

## Summary

- Add a small, unit-tested `restore_has_headroom` predicate for multi-device cache restores.
- Before restoring an incoming conversation snapshot, sample free VRAM on the primary CUDA device and every layer-split stage.
- If any sampled device is below `--vram-reserve-mib`, return the snapshot to the conversation cache and reread the prompt instead of risking a process-fatal CUDA allocation.
- Preserve the existing behavior when no device sample is available or a CUDA memory query fails; this is a conservative fallback, not a new hard failure path.

## Evidence from #2408

A Strata 0.1.40 run on an RTX 3080 Ti + Tesla V100-PCIE-32GB used a two-GPU layer split with a 700 MiB VRAM reserve and a 4096 MiB conversation cache. The log recorded:

```text
strata serve: conversation cache: parked 148167 tokens in 1573.2 ms; parked=2 bytes=3335695744 evictions=8 snapshot_bytes=3097849640
strata serve: conversation cache: restored 148167 tokens (live) in 524.1 ms; parked=2 bytes=2737274704
strata serve: prompt 148295 tokens = 148167 reused + 128 read in 3229 ms (39.6 tok/s), 88 generated in 11781 ms (7.5 tok/s), drafts accepted 54 of 61, 6 checkpoints
CUDA error: out of memory
  current device: 1, in function ggml_cuda_set_device
```

The restore path is an optimization; when the later split stage is already below the configured reserve, rereading the prompt is safer than admitting the snapshot and failing during device residency reconstruction.

## Validation

Passed with the local CPU-only test configuration (the tests cover the shared restore-admission helper):

- `cmake -S C:/tmp/strata-pr-repo -B C:/tmp/strata-pr-repo/build-cpu -G Ninja -DSTRATA_ENABLE_CUDA=OFF -DSTRATA_BUILD_TESTS=ON -DCMAKE_BUILD_TYPE=Debug`
- `cmake --build C:/tmp/strata-pr-repo/build-cpu --target split_rules_test conv_cache_test --parallel 4`
- `ctest --test-dir C:/tmp/strata-pr-repo/build-cpu -C Debug --output-on-failure -R "^(split_rules_test|conv_cache_test)$"` — 2/2 passed
- `git diff --check`

Mehr auf der Site

Links zu Install, Modellen, Releases.