#6436 [Engine] Park the engine: give all VRAM back between requests, keep the process, the context and the conversation
open · @ralfhartmann · 0 commentaires · Sur GitHub
BenchmarksServer & APINVIDIA / CUDAModels & quantsWindows
Description
**What** A way to hand the whole card to another program for a while and get Strata back in seconds, without unloading: - `POST /v1/park`: between requests, the engine copies what it must keep to pinned RAM and frees its device memory. The expert cache may be dropped, because it refills from RAM. - `POST /v1/unpark`, or the next request: the memory comes back at the **same device addresses**. The K/V, the conversation cache and the captured verify windows are as they were, so there is no reload and no new prefill. **Why** We run one RTX 3090 Ti (24 GB) for Strata (Swift-Qwen3.8-Flash-Next IQ3_S, `--native`, `--spec 4`, MTP, 1M context with `--kv-resident 32768`, int8 K/V, `--vram-elastic`) and, at times, a llama.cpp model or an image job. - Unloading and loading Strata again costs 50 to 58 s per load (round trip 72 to 106 s). - NVIDIA's `cuda-checkpoint` works on the engine: park 16 to 17 s, restore 6 to 8 s, and a 410k-token context survives. But it needs root on the host, an external binary under NVIDIA's licence, and the host PID (an LXC guest cannot do it itself). - `POST /v1/vram` (#533) already gives back the expert-cache segments down to the prompt path's floor, here up to about 14 of 23.4 GiB. The dense weights, the MTP head, the K/V window, the verify buffers and the scratch stay. #4168 (lending part of the cache) is related but does not free the card. **What we measured: an `LD_PRELOAD` proof of concept on the real engine** The ready-made engine (terminal 0.1.61, engine 0.1.60, sm_86) links `libcudart.so.13` dynamically and gets its VMM functions through `cudaGetDriverEntryPointByVersion`. Our preload library routes every `cudaMalloc` of 2 MiB or more into VMM ranges, shadows the engine's own `cuMemCreate/Map/...` (the segmented expert cache) with stable handle ids, closes a gate on launches, copies and GEMMs while parked, and parks/unparks over a socket. After load it manages 143 `cudaMalloc` blocks (7534 MiB) + 29 expert-cache VMM handles (14810 MiB) = 22147 MiB; 189 small allocations (166 MiB) stay resident. Six park/unpark cycles on the GPU: | | pinned copies freed after unpark | pinned copies kept | |---|---|---| | park | 13.2 to 13.5 s (pinned alloc ~4 s + copy ~9 s) | 2.07 s (first park 13.4 s) | | unpark | 4.5 to 4.7 s (copy 2.2 s, rest `cudaFreeHost`) | 2.2 to 2.5 s | | engine VRAM while parked | 836 MiB (from 23180 MiB) | 836 MiB | | host RAM while parked | +22.3 GiB | +22.3 GiB, kept after unpark | | TTFT after unpark | 0.59 to 0.61 s (control without park 0.61 to 0.64 s) | 0.59 to 0.62 s | | code tok/s after unpark | 113 to 118 (control 111 to 118) | 112 to 117 | | GPU errors (Xid/NVRM) | none | none | A real client request (508k prompt tokens, 507k reused from the conversation cache) that arrived while parked waited at the gate, ran after unpark and finished normally. Measured DMA: device to host 11.6 GiB/s into kept pinned memory but only ~2.4 GiB/s into freshly allocated pinned memory; host to device ~10 GiB/s. An identity check of the answers before and after park was not decidable: the engine is not reproducible at temperature 0 with a fixed seed even without park (5 different short answers in 6 runs, differing in the last sentence), as already reported in #4673, #4059 and #5626. The answers after unpark were all among the variants that also occur without park. A proper check would need a deterministic engine path or checksums of the device blocks before park and after unpark. **Proposal** (the engine already has most of the pieces) 1. **One allocator for device memory.** There are about 260 `cudaMalloc` calls in the engine sources (generate.cpp 41, q36_engine.cu 35, prefill.cpp 26, mtp.cpp 18, verify.cpp 16, dflash2.cpp 12, ...). Routing them through one function (`strata::core::dev_alloc/dev_free`) that allocates from a reserved VMM range (`vmm.cpp` already resolves `cuMemAddressReserve/Create/Map`) would make every buffer parkable with a fixed address. A force-included header that maps `cudaMalloc`/`cudaFree` to it would avoid touching the call sites. 2. **A `PARK` / `UNPARK` line in the serve loop**, next to `VRAM`, with the same quiesce as `vram_command` (`ver.wait_commit`, `adapt_quiesce()`, `cudaDeviceSynchronize`, no batch slot decoding, no loan out). PARK resets the expert cache's residency (as a full shrink does), copies the remaining live ranges to pinned RAM, then `cuMemUnmap` + `cuMemRelease`. UNPARK maps again, copies back, and refills the cache from RAM, the way the `VRAM` grow path refills `vram_evicted`. CUDA graphs and kernel arguments stay valid because the addresses do not change. 3. **Server:** `POST /v1/park` / `/v1/unpark` take the same fifo and ctl locks as `/v1/vram`. The status reports `"parked": true`, a request while parked unparks first, the on-demand vision reader is stopped at park, and the silence watchdog ignores a parked engine. Natively, with the expert cache dropped and refilled from RAM instead of copied, park should need only ~8.5 GiB of copies (about 1 s into kept pinned memory) and unpark about 3 to 4 s including the cache refill. **Questions** - Would you take this into the engine, or as an opt-in flag (`--parkable`)? - Are there device buffers outside `cudaMalloc` and the two VMM users (`vmm.cpp`, `expert_cache.cpp`) that we missed, for example in the image or vision paths? We are glad to test a native version on sm_86 and report the numbers.
Sur le site
Liens install, modèles, releases.