Pull requests / #2409
#2409 fix: avoid CUDA OOM during conversation-cache restore
open · @MarsLuay · 0 Kommentare · Auf GitHub
BenchmarksServer & APIMulti-GPUNVIDIA / CUDA
Beschreibung
Fixes #2408 ## Summary - Add a small, unit-tested `restore_has_headroom` predicate for multi-device cache restores. - Before restoring an incoming conversation snapshot, sample free VRAM on the primary CUDA device and every layer-split stage. - If any sampled device is below `--vram-reserve-mib`, return the snapshot to the conversation cache and reread the prompt instead of risking a process-fatal CUDA allocation. - Preserve the existing behavior when no device sample is available or a CUDA memory query fails; this is a conservative fallback, not a new hard failure path. ## Evidence from #2408 A Strata 0.1.40 run on an RTX 3080 Ti + Tesla V100-PCIE-32GB used a two-GPU layer split with a 700 MiB VRAM reserve and a 4096 MiB conversation cache. The log recorded: ```text strata serve: conversation cache: parked 148167 tokens in 1573.2 ms; parked=2 bytes=3335695744 evictions=8 snapshot_bytes=3097849640 strata serve: conversation cache: restored 148167 tokens (live) in 524.1 ms; parked=2 bytes=2737274704 strata serve: prompt 148295 tokens = 148167 reused + 128 read in 3229 ms (39.6 tok/s), 88 generated in 11781 ms (7.5 tok/s), drafts accepted 54 of 61, 6 checkpoints CUDA error: out of memory current device: 1, in function ggml_cuda_set_device ``` The restore path is an optimization; when the later split stage is already below the configured reserve, rereading the prompt is safer than admitting the snapshot and failing during device residency reconstruction. ## Validation Passed with the local CPU-only test configuration (the tests cover the shared restore-admission helper): - `cmake -S C:/tmp/strata-pr-repo -B C:/tmp/strata-pr-repo/build-cpu -G Ninja -DSTRATA_ENABLE_CUDA=OFF -DSTRATA_BUILD_TESTS=ON -DCMAKE_BUILD_TYPE=Debug` - `cmake --build C:/tmp/strata-pr-repo/build-cpu --target split_rules_test conv_cache_test --parallel 4` - `ctest --test-dir C:/tmp/strata-pr-repo/build-cpu -C Debug --output-on-failure -R "^(split_rules_test|conv_cache_test)$"` — 2/2 passed - `git diff --check`
Mehr auf der Site
Links zu Install, Modellen, Releases.