#1853 Feature Request: API for temporary VRAM offloading (sleep/wake endpoints) for single-GPU orchestration

本文

TL;DR
Requesting a pair of HTTP endpoints — POST /release_memory_occupation and POST /resume_memory_occupation — that would allow Strata to gracefully free GPU VRAM (by paging weights to Host RAM) without killing the process. This would enable zero-restart, low-latency GPU handoff between Strata and other engines (like ComfyUI) on a single GPU, dramatically improving UX for single-GPU workstations.
Context & Problem Statement
I run a local AI stack orchestrating two engines on the same GPU (RTX 5090, 32 GB):
ComfyUI for image/video generation (LTX-2.5, Qwen-Image-Edit, etc.).
Strata for LLM inference (planning to migrate to Strata + Qwen3.8-Flash-Next-125B).
My manager service arbitrates the GPU. Before a ComfyUI job, it needs the LLM engine to sleep. When SGLang is used, this relies on its memory-occupation API (release_memory_occupation), which takes under 5 seconds because it moves weights to host RAM.
Currently, Strata has no equivalent. The only way to free VRAM for ComfyUI is to kill the process and restart it. While cold starts have improved massively in v0.1.40/41 (~70s for Q2_0 262K context, thank you!), a process restart is still an order of magnitude slower than a RAM-to-VRAM wake cycle. In reverse handoffs, an external report measured ~5 minutes to fully hand the GPU back to an LLM engine using process-level control.
On a 32GB card, this is the difference between "chat feels instant" and "chat feels like a batch job". Tools like T8mars/Comfyui-Strata-T8 exist but rely on process killing, meaning every transition costs a full cold start.
Proposed Solution
Add optional endpoints to the Strata server (e.g., gated behind STRATA_ENABLE_MEMORY_OCCUPATION_API=1):
1. POST /release_memory_occupation
Behavior: Move all resident hot-experts, KV-cache, and vision encoder weights from VRAM to Host RAM (not disk). Keep the process alive. Flush the CUDA allocator pool so nvidia-smi shows freed VRAM. Block incoming inference requests with 503 Service Unavailable.
Optional Payload: Allow surgical offloading.
{
  "offload_targets": ["experts", "vision", "kv_cache"],
  "keep_in_vram_mib": 2048
}
Return: 200 OK once VRAM is freed (or 503 if it cannot be freed within a timeout).
2. POST /resume_memory_occupation
Behavior: Move weights back from RAM to VRAM. Recompute the optimal resident-expert set. Unblock inference.
Return: 200 OK once ready to serve.
3. GET /memory_occupation_state (Optional but helpful)
Return: {"status": "parked", "vram_used_mib": N, "ram_pinned_mib": N, "ready_for_inference": false}. Allows orchestrators to make decisions without scraping nvidia-smi.
Technical Considerations / Implementation Ideas
Since Strata's core architecture already handles hot/cold expert paging, this shouldn't require rewriting the inference engine. It could be implemented as a "forced parking" mode over the existing memory manager:
Pinned Memory: Allocate a pool of Page-Locked (Pinned) Memory in Host RAM at startup. Using DMA for PCIe transfers could drop the offload/restore times for a 125B MoE model to just a few seconds on PCIe 4.0/5.0.
MoE Manager Hook: A global is_parked flag that intercepts the scheduler, forcing all hot experts to be marked as cold and paged to the RAM pool.
KV-Cache: Either copy KV blocks to RAM, or optionally quantize them on the fly (e.g., FP8/INT8) to halve the PCIe transfer time before parking.
Alternatives Considered
Process-level handoff (Process Kill): Works (e.g., T8mars nodes), but every transition triggers a 70s+ cold start.
Reduced Residency (--vram-reserve-mib, --resident-experts): Helps for light ComfyUI workloads, but breaks down completely when ComfyUI needs 20+ GB for video generation models.
Two GPUs: The correct long-term answer, but not an option for the vast majority of single-GPU users.
OS-level SIGSTOP: Doesn't work because the NVIDIA driver holds the CUDA context and VRAM allocations until the process dies or explicitly calls cudaFree.
Environment
Hardware: RTX 5090 (32 GB VRAM), 128 GB Host RAM
OS: Linux (Ubuntu 24.04)
Strata Version: v0.1.41
I am more than happy to test a beta build, provide detailed timing numbers, or help profile the PCIe transfer bottlenecks if this aligns with your roadmap. Thanks for the incredible work on v0.1.40/41 — the CPU-assisted prefill and multi-GPU concurrency improvements are excellent!