#4310 [Engine] session files: format v2 with per-stage save/restore under --layer-split

open · @sakuumomo · 0 comentarios · En GitHub

BenchmarksSetup & installServer & APIMulti-GPUAMD / HIPNVIDIA / CUDAModels & quantsDocumentation

Descripción

Problem and result: Session SAVE/RESTORE works today, but a save of an engine running with `--layer-split` (stages spread over devices) cannot capture or admit the split state: SAVE skipped stage ownership and RESTORE refused the file outright. After this change SAVE captures and transfers each split stage on its own device, RESTORE works under `--layer-split` with per-stage limits and the reclaim hook, and the on-disk format becomes v2 with an explicit stage table. Single-device saves carry a fixed **+24-byte payload delta** (8 B stage-table count + 8 B per checkpoint record; the 64-byte header is untouched apart from its `version` field), and v1 files are **not** read: a v2 engine refuses them as `unsupported version 1` (typed `invalid`), so slot files written by a pre-v2 engine are re-saved once after the upgrade. (Corrected 2026-11-07 per review; see the changes section.)

Dependencies: none — this branch is `main` + 18 commits (re-based state; see the 2026-11-20 changes section) and stands on its own.

Related discussion: none filed; this is the engine groundwork the serve session-ladder PR (#4306) builds on for multi-device hosts.

Approach: format v2 adds a stage table to each checkpoint, snapshot sources and read limits per stage, per-stage validate/restore, a reclaim hook for RESTORE's read admit, then the SAVE/RESTORE paths themselves, a review-driven regression battery, documentation, per-stage device capture/transfer, size-bound fixes, and the main carve following the engine's primary device. (The individual commit hashes of earlier lineages are dropped here on purpose; each rebase renumbered them.) Why a stage table: v1 assumed one device owns the whole conversation cache, so a split save silently lost per-stage ownership at restore. Compatibility (corrected 2026-11-07): v2 is a hard version bump. A v2 engine refuses v1 files (`unsupported version 1`, typed `invalid` — pinned by this PR's own test), and a v1 engine refuses v2; there is no migration path or format knob, so existing `.bin` slot files are re-saved once after the upgrade. Single-device saves differ from v1 by +24 B of payload (8 B stage-table count + 8 B per checkpoint record); nothing is added to the 64-byte header, and `version`, `payload`, and both hashes all change, so the files are not byte-identical in any sense.

Validation:
  Passed: engine compiles and the repo's session-save reclaim test target passes on the HIP build (RX 6900 XT, momo) on the owner's fleet tree carrying this exact content (deploy lineage `54d6c213` era, and the current `nixos` branch rebuilt onto upstream `v0.2.8` — where the session test family passes 7/7: append 1969 checks, file 263, watermark 21, plus a ThreadSanitizer run of 1969 + 260 checks with 0 warnings); live two-card SAVE/RESTORE battery on momo passed (M3 battery records, 2026-10-24: split save/restore round-trips verified end to end); the serve suite is green on the fleet tree containing these commits.
  Failed: none observed on the HIP build.
  Not run: CUDA-only and SYCL builds of this exact rebased branch — the identical content compiles in the owner's hybrid CUDA+HIP fleet build (RTX 4060 + Radeon 890M), but a pure-CUDA CI build of this branch was not executed; hardware execution on NVIDIA-only hosts is not claimed.
Limits: v2 is exercised on AMD (HIP) and hybrid CUDA+HIP hosts only; `--layer-split` stage capture covers the devices the engine itself exposes, and per-stage bounds follow the existing state bounds (no new budget knob).

## Changes since the last update (2026-11-07)

- Rebased onto upstream `v0.2.12` (`123bad4a`, engine 0.1.58): zero conflicts, and the diffstat reproduces exactly (+1291/−69) — the same reproduction @nazerim reported from their independent 877-commit rebase. New tip `6c91c00a` (13 rebased commits + four review follow-ups below), mergeable, metal-link and title jobs green.
- Body corrected per review §2: "+8 header bytes" replaced by the real +24 B payload delta (8 B table count + 8 B per checkpoint record), and the "v1 files still restore" claim removed — the reader has always refused `version != 2`, and this PR's own test pins that refusal.
- Review follow-ups carried on this branch:
  - §3: `conversation_snapshot_sources_split` joins the background GDN copy (opt-in `STRATA_CKPT_ASYNC`) — once after the live capture, and per stage part before it moves into the record; the last part's copy used to be covered only by the *next* stage save's entry join, and with no next save a torn stage could land in a payload-hash-valid file. The battery's own three unjoined comparisons are joined too, the `STRATA_TEST_BIG_GDN` geometry hook is added, and `conversation_stage_split_test_ckpt_async` is registered as a ctest variant — reproducing the reviewer's finding and its proof independently.
  - §4: the SYCL port now defines the 7-parameter `conversation_kv_source` the shared header declares (its 6-parameter definition left the port with an undefined symbol at link), and its `conversation_session_read_limits` engages an empty stage table with `max_stages = 0` — closing the open read bound and restoring the stage-table comparison the PR claims. Not compiled here: no SYCL toolchain, and no SYCL CI job, which is how both halves survived.
  - §9 + Minor: `session_read_reclaim` takes `(chain, cache, …)` like its sibling `session_save_reclaim` (engine call site and battery updated); the header note and the flag's docs line record that RESTORE's ladder runs **before** the stage-table comparison — a file written by another split spends retained K/V, unpinned parks and unpinned checkpoints, and is refused only afterwards. The re-validation fix (or a header-level split check before the RAM admit) remains open.
  - §6–§8 docs: the "pin the split" line for stable files across boots (`--layer-split N`, the K from the startup log's `layer split auto: K=…`; a manual value also keeps the server's per-boot `ordered_gpus` reorder from moving a stage to another card); the restore RAM peak on `--kv-resident` hosts (`SERR memory 0 not enough RAM to read it` near the limit is the expected outcome, with the two-pass streamed read named as the unimplemented fix); placement continuity (fixed tier ⇒ byte-identical continuation after restore; adaptive-evicting tier ⇒ live and restored turns differ **from each other** while re-saved files stay byte-identical; plus the per-compute-capability arithmetic note and the device set being outside the file's fingerprint); and the stale "session files do not support --layer-split" quote in `bench/results/2026-10-10-community-2x-p40-iq3_s/README.md` corrected.
- The `cudaSetDevice`/`cudaGetDevice` wrap-harness offer (§5) is accepted — to arrive as its own PR. The `devices{0,0}` battery limitation is real: this branch's per-stage device discipline is exactly what such a harness pins, and only a two-GPU host can prove it today.
- Validation refresh on the current lineage: CPU session family 7/7 (`conversation_append_test` 1969 checks, `conversation_file_test` 263, `session_watermark_test` 31, `session_save_reclaim_test` 167); ThreadSanitizer 1969 + 260 checks, 0 warnings; full serve suite 1640 OK / 37 skipped; CUDA 13 nvcc compile (arch 89) of `strata`, `conversation_snapshot_test` and `conversation_stage_split_test` rc=0; HIP gfx1030 flavor build green with installCheck, and the GPU conversation battery runs on a real gfx1030 (RX 6900 XT): all 12 targets build and 11 pass — `conversation_snapshot_test` (4377 checks), both `conversation_stage_split_test` variants (including the new `_ckpt_async`), and the seven core session tests. The lone failure is `conversation_snapshot_test_ckpt_async` at the six-restore free-memory pin: `ckpt_async_on()` is compiled OFF for HIP (and Metal), so on gfx1030 that variant runs the synchronous path with the battery's big-GDN geometry, and there each restore moves more than the card's 4 MiB slack — while the `_ckpt_vram_slots` variant at the same geometry passes and every restore/hash check before the pin passes in both variants. Logged as a HIP-lane follow-up, not a correctness failure. (The battery run also exposed an upstream CMake gap: the standalone conversation test targets register with `-I include` only, no hip_compat wiring, so `session.hpp`'s unconditional CUDA include cannot compile under `STRATA_ENABLE_HIP` without a configure-time include — worked around at configure here, worth its own fix upstream.) The unlike-card hardware half remains with the validating reviewer.

Commits were re-signed (GPG ed25519) on 2026-11-07 after rebasing onto upstream `v0.2.12` (`123bad4a`); trees are unchanged from the originally validated work apart from the four review follow-ups (branch tip `6c91c00a`; pre-rebase tip `62cdf9bc` fetchable as an ancestor of the train tag `pin-f6f00908`).




## Changes since the last update (2026-11-20)

- Rebased onto upstream `v0.2.17` (`12d491da4`); new tip `c69025f2` (18 commits), mergeable against current `main`. The adaptation to #1492: the split siblings now take the draft as `const QsaState*` with a `marked == (draft != nullptr ? 1 : 0)` guard, so a no-drafter split save/restore works the same way the single-session path does. The test registrations moved with your CMakeLists split. One follow-up our HIP gate caught: the split battery had six call sites still passing the draft fixture by value where the siblings take `const QsaState*` — a CPU or HIP-off tree never compiles those targets (they register in `engine/cmake/strata_gpu_engine.cmake`), so it surfaced on the first HIP build; fixed at the tip.
- On the `hip_compat` include-wiring gap @nazerim spotted: with the re-based branch the standalone-conversation registration is your CMake's now, so if the gap still exists on 0.2.17 it is a fix for the tree itself rather than this branch — we would gladly test a patch, but we should not smuggle it through a feature PR.
- Gate refresh on the rebased tree: the CPU session family is 7/7 (`conversation_append_test` 1969 checks, `conversation_file_test` 263, `session_watermark_test` 31, `session_save_reclaim_test` 167, plus the split-failure/cache/memory targets), and the ThreadSanitizer build is green with zero warnings. The HIP conversation battery on a real gfx1030 is 20/20: both `conversation_snapshot_test_ckpt_async` and `conversation_stage_split_test_ckpt_async` are green on this lineage, so the 0.2.12-era HIP-lane follow-up (the `_ckpt_async` free-memory pin) no longer reproduces. The battery's first run had one failure — the fresh-card `conversation_copy_streams_test` measured 146 MiB on the card driving the desktop's display, an amdgpu first-attach scanout artifact of the test host; with a headless GPU enumerated first it measures 8 MiB against its 16 MiB bound (see #3551's 2026-11-20 update). The HIP flavor build from a clean checkout of the new pin is green with installCheck; the CUDA+HIP hybrid build links `strata` plus all 20 battery targets in one process (rc=0).
- No CI checks were reported on the new head; the PR is mergeable against current `main`.

Commits were re-signed (GPG ed25519) on 2026-11-20 after the rebase; trees are byte-identical to the gated unsigned tips (tags `gate0217-*`); the pre-rebase tip `6c91c00a` stays fetchable as an ancestor of the tag `pin-pre0217-pr-session-files-layer-split`.

## Changes since the last update (2026-11-21)

Rebased onto upstream `v0.2.18` (`6e38f588a`); branch tip `c69025f2` -> `b25ee11ed`. 1:1 range-diff against the 0.2.17-era chain: the same eighteen commits, zero conflicts, zero adaptations — the 227-commit upstream release rode almost entirely through `engine/src/hy4flash/`, and the only touched overlap file (`engine/CMakeLists.txt`) replayed cleanly. Gates re-run on the rebuilt lineage `nixos` @ `295872912` (88 signed commits): the CPU session family 7/7 with every check count identical to the 0.2.17 record (`conversation_append_test` 1969, `conversation_file_test` 263, `session_watermark_test` 31, `se

En el sitio

Enlaces a install, modelos, releases.