Pull requests / #3286
#3286 [Engine] session: support multi-GPU layer-split save/restore (format v3)
open · @LuckyKo · 0 commentaires · Sur GitHub
Server & APIMulti-GPUAMD / HIPNVIDIA / CUDAModels & quantsWindowsLinux
Description
## Problem and result When running Strata with `--layer-split auto` across multiple GPUs, `POST /slots/0?action=save` answers 400: "session files do not support --layer-split". The save was either refused or produced an incomplete file that couldn't be restored. This made `--slot-save-path` unusable for the most common multi-GPU configuration, which is exactly where long-context session persistence matters most (the RAM cost of re-reading a 30K+ token prompt). After this PR: save and restore work on multi-GPU layer splits. A 94K-token session saves in ~5.5 s and restores in ~3.7 s after an engine restart (vs. 147 s to re-read the prompt from scratch). ## Related discussion none ## Approach Introduce **format v3** for session files that carries per-stage data. Version 2 is reserved for PR #2247's BF16 state codec; this build rejects it explicitly at the version gate. - **Write:** `put_payload` serializes stage 0 (as before), then — only when `stage_images` is non-empty — a count followed by one block per later stage. Each block holds that stage's running state, K/V parts, and checkpoint. The file header version is set to `kVersionStages = 3`. - **Read:** `get_payload` reads stage 0 identically to v1, then checks the version. For v3 files it reads the stage count (bounded by `limits.max_stages`) and each subsequent stage block, validating geometry and layer range per stage. A v1 file has no stage section; a v2 file is refused at the version gate (not silently misparsed as v1). - **Save guard:** if `conversation_checkpoints_split` routes any checkpoint to `cs.rest` (incomplete for the split), the save is refused rather than silently dropping it. - **Read-limit hardening:** `SessionReadLimits` gains `max_stages` and a per-stage `stage_limits` vector. The serve loop populates these from the live engine's stage carves before calling `session_file_read`, so a file from a different split is refused before any allocation. - **Backward compatible:** v1 files read unchanged. A v3 file opened by old code is refused at the version check. Supersedes #2590 (closed by mass-cleanup, same branch). ## Validation **Passed:** - `build-win-cuda13\conversation_file_test.exe` — 208 checks (rev `94f55cd6`, CUDA 13.4, Windows 11, 2× RTX 5060 Ti sm_120) - `build-win-cuda13\conversation_snapshot_test.exe` — 4027 checks - `build-win-cuda13\conversation_cache_test.exe` — 4219 checks - `build-win-cuda13\conversation_memory_test.exe` — 25 checks - `build-win-cuda13\conversation_split_failure_test.exe` — 36 allocation-failure cases (stages 1–3) - `build-win-cuda13\session_save_reclaim_test.exe` — 119 checks - Smoke test: engine starts on dual-GPU layer-split, chat completion responds, session save/restore round-trip works - Independent test on Linux (2× NVIDIA A2, sm_86, Debian 13): 23K and 94K token sessions, save/restore verified, invalid files properly refused (by @Eranicus30666, rev `75e7df6b`) **Failed:** - none observed **Not run:** - `--batch` / parallel slots (no multi-slot test harness) - Restore with a different GPU split than the save (unit tests cover the refusal path) - Full-disk / near-full-VRAM restore (see #2409) - HIP, SYCL, Metal builds (CUDA-only development environment) ## Limits - Does not support `--batch` (parallel slots) — the stage layout is per-slot and the save format assumes a single active slot. - A v3 file is not readable by pre-0.2.0 engines (refused at the version check, no data loss). - The one-clause reply difference observed at 94K tokens by @Eranicus30666 is decode non-determinism (floating-point accumulation order between a fresh prompt pass and an incrementally-built restored state), not a serialization bug.
Sur le site
Liens install, modèles, releases.