Pull requests / #2119
#2119 Add disk-backed conversation cache parking (--conversation-cache-storage disk)
open · @dsovsyannikov · 0 comments · View on GitHub
BenchmarksServer & APIMulti-GPUNVIDIA / CUDAModels & quantsWindowsLinux
Description
## Relation to parallel work Two parallel efforts in the queue address disk parking; keeping this PR open as the minimal third option, with the differences called out: - **#1937** (older, Oct 11): disk as an **overflow tier under the RAM cache** — an evicted RAM entry is written to disk instead of dropped, via the same session-file primitives this PR uses. Hot conversations stay in RAM; a background writer keeps switches from waiting on the disk; `--conversation-disk-persist` survives restarts. Best when a RAM budget exists and only its overflow is the problem. - **#2098**: a **page-granular disk tier** with version-tagged snapshot directories, a shared blob, GC and 467 checks, field-tested at 62–68 GB. The fullest design if cross-version persistence and page sharing matter. - **This PR**: disk as **the parking backend itself** (`ram|disk`), not a spill target — a parked conversation holds zero resident RAM (measured −26 MiB vs +308 MiB held). Targets hosts where even the RAM parking budget does not fit beside the expert arena (admission there skips every conversation), or wherever you simply do not want parked state in RAM. Single +964/−15 commit; a parked conversation's bytes are identical to what `POST /slots/0?action=save` already writes. Related: #2082 (admission fail-safety — the Windows commit-limit cases this PR's floor handling also dances around) and #2077 (`--prompt-cache-file`, a complementary restart-persistence angle). ## Motivation The conversation cache parks a switched-out conversation as a snapshot in host RAM, which keeps alternating chats (an agent controller plus subagents, for example) from re-reading their history. But on a machine whose RAM cannot hold the resident expert arena beside a parking budget, the admission check (`available_host_bytes()` vs the snapshot estimate + `--conversation-cache-min-free-mib`) skips every conversation, so the feature is effectively unavailable there. This PR adds an opt-in disk backend: parked snapshots live on SSD and cost no resident RAM. ## What it adds - Engine flags: `--conversation-cache-storage ram|disk` (default `ram` — current behaviour unchanged) and `--conversation-cache-dir DIR`. In disk mode `--conversation-cache-mib` is the disk budget. - `ConversationDiskCache` reuses the session-file snapshot format with its validation and atomic publish (temp file + rename + flush), so a parked conversation is the same bytes a `SAVE` writes. LRU eviction deletes the oldest file when the budget is exceeded. - The RAM side keeps only a small index (conversation ids/images): after `park()` the snapshot costs no resident memory. - `/metrics` `conversation_cache` gains a `storage` field; the Monitor tab reflects the mode. - Known limits: disk mode refuses `--layer-split` (the session format has no stage images); retained-K/V reuse stays a ram-mode feature; restore speed depends on the page cache (42.5 ms hot in the measurement below, closer to the session-restore second when cold); a snapshot is tied to the engine version that wrote it (no tag gating — see #2098 for that machinery). ## Tests - New `conversation_disk_test` (55 checks): park/restore byte-exact, LRU eviction, budget accounting, preflight refusal. - `tools/parking_test.py` and `tools/conversation_cache_parity.py` accept `--storage disk`; new `tools/conversation_cache_disk_measure.py` reproduces the numbers below. - Parity gate A/B/A: **byte-exact main-model state** in both backends. - Built and run on Windows 11 (MSVC 19.44 + CUDA 13.3, RTX 5090 Laptop) and WSL2 Ubuntu 24.04 (CUDA 13.3), engine 0.1.43, base 7695859a. ## Measured (RTX 5090 Laptop 24 GB, IQ3_XXS pack, 14.4K-token conversation, 435 MiB snapshot) | metric | ram | disk | | --- | --- | --- | | park | 105 ms | 411 ms (456 MB NVMe write) | | restore | 32 ms | 42.5 ms (page-cache hot) | | Δ engine RSS on park | **+308 MiB held** | **−26 MiB** | | reused tokens on resume | 14087 / 14188 | 14087 / 14188 | | decode after restore | 30.3 tok/s | 27.7 tok/s | Restoring the same conversation from disk after a full engine restart took 152 ms and the continuation was served from the restored state. On the RAM-tight host this targets (resident arena leaves ~1.4–2.7 GiB free), ram mode skips parking entirely while disk mode keeps alternating conversations at ~2–4 s per switch instead of a full re-read of the history.
Related on strata.com
Editorial links to help you install, pick models, or read release notes — not part of the upstream thread.