改动 / #3286

#3286 [Engine] session: support multi-GPU layer-split save/restore (format v3)

open · @LuckyKo · 0 评论 · 去 GitHub 看

Server & APIMulti-GPUAMD / HIPNVIDIA / CUDAModels & quantsWindowsLinux

说明

## Problem and result

When running Strata with `--layer-split auto` across multiple GPUs, `POST /slots/0?action=save` answers 400: "session files do not support --layer-split". The save was either refused or produced an incomplete file that couldn't be restored. This made `--slot-save-path` unusable for the most common multi-GPU configuration, which is exactly where long-context session persistence matters most (the RAM cost of re-reading a 30K+ token prompt).

After this PR: save and restore work on multi-GPU layer splits. A 94K-token session saves in ~5.5 s and restores in ~3.7 s after an engine restart (vs. 147 s to re-read the prompt from scratch).

## Related discussion

none

## Approach

Introduce **format v3** for session files that carries per-stage data. Version 2 is reserved for PR #2247's BF16 state codec; this build rejects it explicitly at the version gate.

- **Write:** `put_payload` serializes stage 0 (as before), then — only when `stage_images` is non-empty — a count followed by one block per later stage. Each block holds that stage's running state, K/V parts, and checkpoint. The file header version is set to `kVersionStages = 3`.
- **Read:** `get_payload` reads stage 0 identically to v1, then checks the version. For v3 files it reads the stage count (bounded by `limits.max_stages`) and each subsequent stage block, validating geometry and layer range per stage. A v1 file has no stage section; a v2 file is refused at the version gate (not silently misparsed as v1).
- **Save guard:** if `conversation_checkpoints_split` routes any checkpoint to `cs.rest` (incomplete for the split), the save is refused rather than silently dropping it.
- **Read-limit hardening:** `SessionReadLimits` gains `max_stages` and a per-stage `stage_limits` vector. The serve loop populates these from the live engine's stage carves before calling `session_file_read`, so a file from a different split is refused before any allocation.
- **Backward compatible:** v1 files read unchanged. A v3 file opened by old code is refused at the version check.

Supersedes #2590 (closed by mass-cleanup, same branch).

## Validation

**Passed:**
- `build-win-cuda13\conversation_file_test.exe` — 208 checks (rev `94f55cd6`, CUDA 13.4, Windows 11, 2× RTX 5060 Ti sm_120)
- `build-win-cuda13\conversation_snapshot_test.exe` — 4027 checks
- `build-win-cuda13\conversation_cache_test.exe` — 4219 checks
- `build-win-cuda13\conversation_memory_test.exe` — 25 checks
- `build-win-cuda13\conversation_split_failure_test.exe` — 36 allocation-failure cases (stages 1–3)
- `build-win-cuda13\session_save_reclaim_test.exe` — 119 checks
- Smoke test: engine starts on dual-GPU layer-split, chat completion responds, session save/restore round-trip works
- Independent test on Linux (2× NVIDIA A2, sm_86, Debian 13): 23K and 94K token sessions, save/restore verified, invalid files properly refused (by @Eranicus30666, rev `75e7df6b`)

**Failed:**
- none observed

**Not run:**
- `--batch` / parallel slots (no multi-slot test harness)
- Restore with a different GPU split than the save (unit tests cover the refusal path)
- Full-disk / near-full-VRAM restore (see #2409)
- HIP, SYCL, Metal builds (CUDA-only development environment)

## Limits

- Does not support `--batch` (parallel slots) — the stage layout is per-slot and the save format assumes a single active slot.
- A v3 file is not readable by pre-0.2.0 engines (refused at the version check, no data loss).
- The one-clause reply difference observed at 94K tokens by @Eranicus30666 is decode non-determinism (floating-point accumulation order between a fresh prompt pass and an incrementally-built restored state), not a serialization bug.

本站相关内容

相关页面的快捷入口。