#5500 [Engine] L2 session cache: cut prefill time by ~95% on returning conversations
open · @giphoo · 0 Kommentare · Auf GitHub
BenchmarksServer & APIMulti-GPUAMD / HIPNVIDIA / CUDAModels & quantsSecurityWindowsLinux
Beschreibung
## Problem and result Long agent conversations that alternate between sessions pay repeated prefill costs when the conversation is no longer in the RAM cache. Existing session files write and read whole snapshots, which amplifies I/O when only a small part changes or only a prefix is needed. **Measured effect** (production traces on a 48 GB AMD card, int8 KV, V3 archives): - cold prefill on this class of card runs ~2.2-2.6K tokens/s — a 145K-token conversation context costs ~56-66s of prompt compute every time it returns; - with the L2 cache, the same returning context prefill-costs **1.6-2.4s** (99.9% reuse, TTFT 5-18ms) when parked in RAM; - after an engine restart, the disk tier restores the archive at **35-88K tokens/s** (205K tokens in 4.0s, 117K in 1.3s, 19K in 0.31s); - net: **>95% of prefill wall time saved** on returning conversations (disk path), ~98%+ when RAM-parked. This PR adds: - a block-based v2 session format (stored blocks only — LZ4 was measured net-negative on int8-quantized KV: ~1.2-1.4x on near-incompressible int8 KV, the CPU cost competes with expert prefetch, so it was removed early); - incremental saves that reuse unchanged checkpoint and K/V segments; - checkpoint-prefix restore into the existing PARKED RAM cache; - an asynchronous writer with per-file generation commits; - a read-only probe for cache availability; - coarse-generation V3 archive directories: manifest + immutable base + deltas + counts + an independent checkpoint, so an incremental save is a cheap append and growth stays bounded by consolidating the delta tail instead of falling back to a full save; - a minimal reference controller and rotation benchmark. All features are opt-in and can be turned completely off: unset `STRATA_SESSION_V2`/`STRATA_SESSION_V3` and the engine never runs this code path — no session files are written in the new formats, existing archives on disk are ignored rather than read, and the reference controller/benchmark only do anything when explicitly invoked. With both switches off, the one remaining behavioral delta is the checkpoint-tolerance scoping noted under Limits. Neither switch is set by any preset, so default behavior is unchanged. ## Related discussion - #57 (multi-conversation caching, solved by #189 for RAM) - #143 (OpenClaw agent full-prompt rereads) - #190 (disk tier PR, closed pending rebase) - #3027 (disk durability for parked conversations) - #3485 (incremental disk-tier measurements) - #4021 (tail-only parking, complementary RAM optimization) - #6081 (capacity-refusal observability; the discussion under this PR in November measured the same wall from the outside) - #6142 (the policy layer that drives these primitives: bytes-only budgets, cost-gated restore, fail-closed admission — draft stacked on this PR) ## Approach The engine owns all KV state and transfers. The controller only decides when to save and restore. The key primitives are: 1. **SAVEINC**: plan against the old file, capture only fresh state, append in place, fall back to full save on any mismatch. 2. **RESTOREP**: stream a v2 file directly into a PARKED entry (no GPU transfer, no active-session replacement). 3. **RESTOREPP / preload**: read only the checkpoint prefix that matches the incoming prompt, as a PARKED entry. 4. **PROBE**: read-only query of how much the cache would return. 5. **WriterPool**: bounded async append queue with per-generation commit tracking, synchronous fallback on budget exceeded. 6. **V3 directories**: each chain is a directory; saveinc appends a delta segment and, at a threshold, folds the tail chain into one consolidated delta from the immutable base; composite readers merge base/delta/checkpoint and keep parallel pread across segments; the manifest is the single commit point. A minimal reference controller (~300 lines, `l2/reference_controller.py`) demonstrates the full save → preload → generate → saveinc loop. A rotation benchmark (`l2/bench_rotation.py`) measures reuse rates across a three-conversation workload with engine restart. ## Validation **Passed (reference implementation, Linux/HIP, AMD W7900 48 GB, Qwen3.8-Flash-Next UD-IQ4_XS, v0.2.12/engine 0.1.58):** - 36/36 L2 contract tests - 5/5 core C++ tests (conversation_cache/memory/file/reclaim/cache_l2) - 99/99 upstream server tests (tool calls, request accounting, Monitor) - 51-request three-conversation rotation: 0 LOW-REUSE, 0 restore 400/404, 0 OOM/segfault - 9 restores, P50=3.41s, P90=7.48s - Writer final pending=0, 5 save commits committed **Passed (branch head `082ad02e`, rebased onto v0.2.15; Linux/HIP):** - pristine configure + full engine target build against v0.2.15 (release config, gfx1100) — zero-conflict rebase, the branch merges cleanly into current main - the rebased branch is a direct distillation of the production line, so it carries one month of fixes since the first snapshot: V3 archive maturation, unique-inode budget accounting, reentrant decision lock, in-saveinc tail consolidation, small-borrow planning, saveinc rescue fallback; the conversation suite (359 checks on the predecessor snapshot, extended since) runs green in that production line **Failed:** - HIP flag-order test on gfx1201 (device enumeration order); passed with `HIP_VISIBLE_DEVICES=1` on W7900 **Not run:** - CUDA build or execution - SYCL build or execution - Metal build or execution - Windows build or execution - Power-loss / SIGKILL recovery for SAVEINC - Multi-GPU / layer-split / peer-device / batch configurations ## Limits - In-place SAVEINC on a v2 single file is not power-loss atomic. A crash mid-append can leave the file unreadable. Recovery needs a journal, dual footer, or COW design. V3 directories are safer here: the manifest is the single commit point, so an interrupted delta append just discards that generation. - File I/O uses POSIX APIs directly (`open/pread/pwrite/fsync`). Windows needs a separate implementation. - The default checkpoint tolerance is currently enabled globally in `conversation_snapshot_restore()`; it should be scoped to the L2 opt-in path only. - Writer memory accounting approximates metadata as 64 MiB; fresh checkpoints may exceed this. - The reference controller is a demonstration, not a production gateway (no auth, no CORS, no scheduler). The production gateway it sketches lives in #6142. - No message-content retention is included. The engine-side format layer is complete and green end to end in production use. Ready for review — and the split is no longer hypothetical: the policy layer (gateway with bytes-only budgets, cost-gated restore, fail-closed admission) is up as #6142, stacked on this PR.
Mehr auf der Site
Links zu Install, Modellen, Releases.