Issues / #3027
#3027 serve: disk durability for parked conversations (saved conversations)
open · @whitespades · 0 comentários · No GitHub
BenchmarksSetup & installServer & APIAMD / HIPNVIDIA / CUDAModels & quants
Descrição
**The ask**: make the conversation cache durable - write each parked conversation's state snapshot to disk (opt-in path) and load it back at start-up, so an engine restart costs seconds instead of a full prompt re-read. **Why it matters (live setup: RTX 3090, IQ3_S, 262k context, engine 0.1.45)**: an agent client (DSH) is stateless and resends the conversation every turn; conversations run 100-140k tokens. Parking + restore works superbly: 131k tokens restored in ~1 s, 93% of all prompt tokens reused across 30 requests, 1 conversation parked. But the snapshot is RAM-only and dies with the engine. Every config change, engine update, reboot pays a full read: `prompt 102104 tokens = 0 reused + 102104 read in 51247 ms (1992 tok/s)`, and 62 s for 132,828. Long agent sessions start from zero after every restart. **Why it fits Strata**: the pieces already exist. Parking already serializes exactly the state that matters (KV block tier, GDN conv history and recurrent state, position, live tokens + image keys), and the KV streaming tier already moves whole blocks between VRAM and system RAM. The durable version swaps the RAM arena for files - the match rules (live tokens, image keys, min-tokens) can stay as they are. **Prior art**: llama.cpp's slot-state save/load, shipped as "Saved conversations" in professorpalmer/mirai-s-ada (bundle-20261017b): with `MIRAI_SLOT_SAVE_PATH` set, a 245k-token conversation is back in ~10 s instead of a 524 s re-read, a 128k document answers in 6.7 s after a restart (199 s cold), the saved K/V (VRAM part + pinned-RAM part + draft state) reloads in seconds and reads only the new turn, same answers byte for byte; the file set is pruned above 20 GB. vLLM's prefix-cache CPU/disk tier follows the same shape. **Rough sketch**: - Opt-in in the config's args: `--conversation-cache-save PATH` (save at park and/or on shutdown; a `--conversation-cache-save-max-gib N` quota pruning the oldest files, as the 20 GB rule does). - File content: the parked snapshot (int8 KV blocks, GDN history/state, position, live tokens, image keys) plus a model/engine fingerprint so a stale or mismatched file is refused at load. - At start-up: warm the slots from disk in the background; a request whose prompt extends a saved prefix restores from disk like from a park (the "1 s for 131k" path). - Size on this family: int8 KV is cheap - 12 attention layers x 2 x 256 B/position ~ 6 KB/position, plus a few MB of GDN state per conversation: 131k tokens ~ 0.8 GB, 262k ~ 1.6 GB; NVMe load ~1-2 s. Fits the RAM budget the conversation cache already uses. **What it does not solve** (worth stating so expectations stay right): a prefix rewrite. A client that rewrites its history at the head of the conversation invalidates every stored K/V after the change point (the vectors encode the old content), and no storage tier changes that - the spread prompt cache covers shallow rewrites; durability covers restarts. The two features are complements.
No site
Links install, modelos, releases.