#6034 [Engine] Parked conversations spill to a disk tier instead of being dropped
open · @nrtdadmin · 0 comments · View on GitHub
Server & APIAMD / HIPNVIDIA / CUDASecurityLinux
Description
Replaces #5399 (format-closed; GitHub would not reopen a closed fork PR, per the bot's suggestion - same branch, now with the erase fix from #5399's keithm999). ## Problem and result The parked-conversation cache only had RAM: once `--conversation-cache-mib` or the slot count filled, `evict_oldest()` **dropped** the oldest conversation and its next request re-read the entire prompt. On our 4x V100 box (long multi-hundred-KB prompts) that re-read is the expensive part, and the eviction happened exactly when memory was under pressure. With `--conversation-cache-file-mib > 0` (new, plus `--conversation-cache-file-dir`, default `<pack>/conv-cache`) eviction now **spills** the heavy parts (K/V pages, running state, checkpoints) to `<dir>/<serial>.conv` and keeps a small RAM shell (ids, images, pinned) so `best()` matching still finds it; a returning conversation is read whole back from disk and the file goes. `0` (default) is upstream behavior byte-for-byte: eviction drops. ## Related discussion #4121 (this box's thread). Independently measured and requested by community users on this PR: @satwareAG-ironMike (RTX 3060, 130-146K agent sessions: a 3.4 GB spill is a 1-2 s read against a 130-190 s prompt re-read, ~100x) and @keithm999 (R9700 HIP). No maintainer sign-off yet - opened for review. ## Approach - A shell still holds one of `slots`: the count bounds RAM and disk entries together; when a slot is needed and the oldest entry is already on disk it leaves for good, same when the disk budget is exhausted. - Budget-pressure eviction spills before it kills a shell (f08c6c2e9): the first version's eviction loop killed the shell the previous park had just written, so the folder held at most one file under any pressure. - `borrows()` refuses a spilled entry - the restore is all-or-nothing, a half-borrowed K/V never reaches the graph. - Reads run through the existing `conversation_snapshot_validate` gate; a half-written file (crash between write and rename - the write is tmp + rename) is refused, not trusted. Enabling the tier wipes the folder. - `take()` deleted the spill file before reading it (245d15bc8, found by @keithm999): every restore failed silently behind a `disk_reads` bump that counted the failures; erase now runs after the read and the counter counts only restores that returned a conversation. - Serve web monitor: a "Their storage (disk)" row under "Their memory (RAM)" showing usage / spilled / restored, hidden when the tier is off. ## Validation **Passed:** - `conversation_cache_disk_test` (new, in the `STRATA_BUILD_TESTS` block): round-trips a 751 MiB synthetic parked image (100k tokens, 5 K/V pages, 3 checkpoints, 2 stage images, a buffer crossing the 16 MiB segment edge), byte-identical; refuses a truncated file; survives an empty image. Revision 245d15bc8, Linux/CUDA12 x86_64 + ASan/UBSan standalone. - Live on the 4x V100 (Volta sm_70, CUDA 12) `--serve` box, a full loaded day after the eviction-order fix: RAM cache pinned at 127/128 GiB with 469 parked conversations, 317 spill files written, evictions down to 25 (from 334 with the bug). - Community reproductions on this PR: keithm999 on v0.2.13 + HIP - with the erase fix, 19 of 22 returning sessions restored from disk, median TTFT 84.7 s -> 9.9 s (before the fix 0 of 18). **Failed:** - The first revision's eviction order under RAM pressure (shell-killing loop), found by live load test, fixed in f08c6c2e9; the erase-before-read bug above, found by keithm999, fixed in 245d15bc8. **Not run:** - HIP/SYCL/Metal builds (no AMD/Intel hardware here) - keithm999's HIP box is the only non-CUDA validation so far. ## Limits - A shell still occupies a slot, so spilled entries count against `slots`; very deep histories may prefer a larger `slots` together with the tier. - The spill file format is engine-private (version-tagged, wiped on enable); no cross-machine portability promised. - On zram/swap-heavy Linux boxes the kernel already gives an implicit compressed overflow tier - the file tier wins on persistence across restarts and engine-controlled accounting (per @satwareAG-ironMike's measurements on this PR). *Built and tested by the owner's ops agent (hence the `[email protected]` commit author). Happy to re-run anything on request.*
Related on this site
Quick links to related pages.