#4551 [Engine] session-file v3: incremental append saves under --session-save-append

open · draft · @sakuumomo · 0 comments · View on GitHub

Setup & installServer & APIMulti-GPUAMD / HIPNVIDIA / CUDAModels & quantsSecurityWindowsLinux

Description

Problem and result: saves rewrite the whole conversation file every time — a 30k-token session costs ~1.1 GB written per save. This adds an opt-in incremental mode: with `--session-save-append`, a save APPENDS its new generation group onto its own file instead of rewriting it, behind the trailer chain + digest authority cascade that already guards a torn tail. Flag-off behavior is unchanged: a flag-off save is byte-identical to v2 (pinned by `conversation_file_test`'s golden fixture).

On a live serve host (proof tables collapsible below): re-save with no request in between went from writing **1,109,307,968 bytes in 8.88 s** to appending **237,226,496 bytes in 1.97 s**; a second shape: 1,053,068,864 / 31.5 s → 237,210,624 / 2.11 s.

Related discussion: #4552 (the steady-state blocker: engine-internal mount/borrow watermarks dirtied from token 0 — **fixed on this branch** as of 2026-11-07: every proven-bytes resume now marks the restored prefix, see Approach and Limits). Stacks on #4310 — see Limits.

Approach:
- v3 format: per-generation groups appended to the file itself (no temp, no rename); header/trailer chain, region-map reader, digest authority + demotion cascade behind a torn tail.
- The save gate (`session_save_gate` + the kv watermark M1–M7 in `SessionState`): a save may append only when the pool's first dirty cell, rounded to whole pages, is at or above the file's **whole-page committed prefix**. The whole-page rounding is load-bearing: comparing against the raw `saved_tokens` made every save at an unaligned boundary refuse its own next append — that bug was found by measurement on a live host (13 saves, 0 appends) and is pinned here with an unaligned gate matrix (`conversation_append_test`, `session_watermark_test`).
- The append reports its DELTA as `bytes` (appended-only SAVED field) so the serve-side ladder can record `appended` / `delta_bytes` in its save ledger.
- Compile portability: the v3 reader held its layer count in `size_t` while the reader's `count(uint64_t& …)` binding needs `uint64_t` — identical types on Linux, `unsigned long` vs `unsigned long long` on macOS arm64, where the reference refuses to bind (`metal-link` CI caught it). The count now follows the file's u64-counts convention; the other twelve `count()` sites already did.
- #4552 fix (added 2026-11-07, in the shape the issue thread converged on): engine resumes that rewrite pools **from proven bytes** mark the restored prefix instead of dirtying from token 0. The core backstops `conversation_kv_restore` / `conversation_kv_restore_prefix` (M9) mark a COMPLETED restore at `upto`; a mid-transfer failure marks 0 (partial bytes landed); a refusal that wrote no pools marks nothing, so a rejected attempt cannot invalidate the watermark a previous save earned. The engine sites follow: the file restore (M3) at the restored conversation's whole cells, a park restore (M4) per carve against its own stage image, a prefix borrow (M4) at the borrowed count, a batch slot handing its conversation back (M5) at the tokens it returned; the zeroing sites (M2) keep marking 0 — there nothing was restored — and the checkpoint mount (M8) already marked at spare-row granularity, unchanged. On a layer split the save gate now consults `session_gate_watermark`, the fold (most-dirty carve) over every carve — a stage rewritten below the committed boundary refuses the append, the file-corrupting direction of the same defect — and the post-save raise lifts every carve's watermark, so a split save stops refusing its own next append.

Validation (branch tip `f1dbcea0` — the append delta re-applied on the rebuilt 0.2.8 lineage; the v3 content is identical to the gate-verified `d74bcec6` tree; local NixOS x86_64, AMD gfx1030 / ROCm 7.2.3; model Qwen3.8-Flash-Next IQ3_XXS GGUF, `--kv fp16 --max-context 262144 --kv-resident 32768 --layer-split 19`):
  Passed:
  - `cmake --build build && ctest --test-dir build -R 'conversation_(file|append|split_failure|cache|memory)_test|session_save_reclaim_test|session_watermark_test'` (nix develop toolchain): 100 % tests passed out of 7. Running the binaries directly: `conversation_append_test` 1969 checks, `conversation_file_test` 263, `session_watermark_test` 21.
  - ThreadSanitizer build: `conversation_file_test` + `conversation_append_test` — 0 warnings (1969 + 260 checks; 3 filesystem-shape file checks skip under the sanitizer).
  - Serve-side python suites: session-ladder 112 tests OK, serve 156 tests OK.
  - Both package flavors build with installCheck on `d74bcec6`: the HIP/gfx1030 build and the CUDA build that also compiles the shared kernels for gfx1150 — rc=0.
  - GitHub Actions on `d74bcec6`: Metal build (macOS arm64), Minimal serving runtime, Title area — all green; on the rebased tip `f1dbcea0` the Metal and title jobs are green again. Metal is compile-only here; no Metal execution claimed.
  - Hardware execution (live serve host running the deployed `dd4790a7` build — the same lineage; the one commit on top is the macOS type binding above): the append/refusal/restoration behaviors in the collapsible tables below, from real saves against a running GPU engine, with the append delta round-tripping through the engine result into the serve-side save ledger.
  - 2026-10-30 rebase onto upstream `v0.2.8` (`870aedd9`): the 16-commit append delta re-applied clean on the rebuilt chain of #4310 + #4306 + #4307 + #4411 + #3551/#3552/#3553 (`nixos` == branch tip; 85 signed commits on `870aedd9`). Gates re-run on the rebuilt tip: session test family 7/7 (append 1969, file 263, watermark 21), ThreadSanitizer 1969 + 260 checks 0 warnings, full serve suite 1386 passed / 17 skipped, HIP and CUDA+HIP hybrid flavor builds rc=0 with installCheck.
  Failed: none on the submitted tree. (The previous head `dd4790a7` failed the macOS `metal-link` compile — that failure is the compile fix above and the current head is green on the same job.)
  Not run: Windows builds; execution on NVIDIA hardware (the CUDA-flavor evidence is compile-only); Metal execution.

  Update (2026-11-07): rebased onto upstream `v0.2.12` (`123bad4a`, engine 0.1.58) with zero conflicts; the append delta re-applies 1:1 and the #4552 fix lands as described in Approach. The branch is fast-forwarded to the rebuilt train tip `c3e4cdc4` (93 signed commits: the seven other PR chains of this stack re-merged in order onto `123bad4a`, the append delta + #4552 fix riding the tip). Gates on that lineage: session test family 7/7 (`conversation_append_test` 1969 checks, `conversation_file_test` 263, `session_watermark_test` 31 — the battery grew with the #4552 pins: completed-restore marking at `upto`, mid-transfer failure marks 0, refusal marks nothing, park/borrow/batch prefix pins, the split-gate fold, the post-save raise of every carve —, `session_save_reclaim_test` 167); ThreadSanitizer 1969 + 260 checks, 0 warnings; full serve suite 1640 OK / 37 skipped; CUDA-13 nvcc (arch 89) compile of `strata`, `conversation_snapshot_test` and `conversation_stage_split_test` rc=0 (GPU execution stays with the validating CUDA hosts); HIP gfx1030 and CUDA+HIP hybrid (gfx1150, CUDA 13) flavor builds with installCheck. Engine identity across this pin (0.1.54 -> 0.1.58; `SessionConfig::engine_version` is in the file's config fingerprint): the first save of each pre-existing session file is a full rewrite by construction.

Update (2026-11-20): rebased onto upstream `v0.2.17` (`12d491da4`); the branch head is the rebuilt train tip `4cfc06018` — 88 signed commits: the seven other PR chains of this stack re-merged in order onto `12d491da4`, the seventeen append/#4552 commits of this branch riding the tip (of the original eighteen, the ladder unit-CFG hygiene fix now rides at the rebased #4306 chain's tip). The #4552 fix (a resume marks the whole-page COMMITTED PREFIX; the split gate folds every carve's watermark) is on top, unchanged from the shape the issue thread converged on. The only non-mechanical conflict was the watermark-M7 site in `generate.cpp`, fused into your deeper nesting unchanged; one one-line adaptation rode #1492: the read-limits split sibling takes the draft as `const QsaState*` (carried in the #4310 chain), so the append gate passes `session_draft`, mirroring the restore-side call. Gates on the new lineage: session test family 7/7 (`conversation_append_test` 1969 checks, `conversation_file_test` 263, `session_watermark_test` 31 — every #4552 pin intact —, `session_save_reclaim_test` 167, plus the split-failure/cache/memory targets); ThreadSanitizer green, 0 warnings; the full serve suite with only the two environment-sensitive non-green entries (#4306's 2026-11-20 update); the HIP gfx1030 conversation battery 20/20; the HIP flavor build from a clean checkout of the pin green with installCheck. Engine identity moves 0.1.58 -> 0.1.63 (`SessionConfig::engine_version` is in the file's config fingerprint): the first save of each pre-existing session file is a full rewrite again by construction. The owner's desktop host was re-pinned to this lineage on 2026-11-20 and runs slot save/restore and the session ladder live; the live-host steady-state re-measure landed on 2026-11-21 (capture on #4552). No CI checks were reported on the new head; the PR is mergeable against current `main`. Commits re-signed (GPG ed25519) on 2026-11-20; trees byte-identical to the gated unsigned tips (`gate0217-*`); the pre-rebase train tip `c3e4cdc4` stays fetchable as the tag `pin-c3e4cdc4`.

Limits:
- Steady-state eligibility was the open question at submission: engine-internal resumes (park-borrow / slot-mount / file-restore) dirtied the pool from token 0, so only saves with untouched pool state in between appended — analyzed with repro in #4552. The #4552 follow-up above closes the proven-bytes half: those resumes mark the restored prefix, so the normal save-after-request cycle is append-eligible again (pinned by the `session_watermark_test` prefix pins). The remaining boundary is unchanged and remains correct: a resume that genuinely rewrites rows below the committed boundary (grow/trim churn) still refuses. A live-host re-measure of the steady state follows the owner's re-pin; the lineage is now `v0.2.17` @ `4cfc06018`, live on the desktop host since 2026-11-20. Re-measured 2026-11-21 on that host: after the by-construction first-save full rewrite (58.1 s, ledger `appended=false`), the next save after an ordinary +22,878-token agent-turn cycle **appended in place** — the 4.93 GB file grew by 891,661,824 bytes in 11.0 s with no temp and no rename, ledger `appended=true, delta_bytes=891661824`, the recorded delta matching the file-size increment byte-for-byte. The steady-state save-after-request cycle is append-eligible again; the grow/trim-below-committed-boundary refusal remains as designed. Full capture on #4552.
- Stacks on #4310 (format v2): left draft until it lands; I will rebase onto `main` then. The merge lineage this branch sits on is fetchable at `sakuumomo/Strata` branch `nixos` (the run-here lineage: #4310 + #4306 + #4307 + #4411 + #3551/#3552/#3553 re-merged in order onto `12d491da4` as of 2026-11-20, the append delta + #4552 fix on the tip; `nixos` == this branch @ `4cfc06018`; earlier lineage tips stay fetchable as tags `pin-*`).

<details>
<summary>Live-host proof tables (click to expand)</summary>

Captured 2026-10-29 against the pre-#4552-fix build (engine 0.1.54 lineage `dd4790a7`). The refusal rows are the real behavior of *that* build — Proof B rows 2–4 show exactly the defect the #4552 follow-up above fixes; a fresh capture against the 0.2.12 lineage follows the owner's re-pin.

Proof A — chain-re-push shape (session `probe-app-e`), engine flags `--session-save-reclaim --session-save-append`:

| save | shape | result |
|---|---|---|
| 1 | fresh file | full 1.109 GB, 8.88 s (`save gate: full rewrite - no verified checkpoint chain for this file`) |
| 2 | re-save, no request in between | **appended** — `n_written` 237,226,496 (delta), 1.97 s, l

Related on this site

Quick links to related pages.