#5835 [Engine] Conversation cache: an opt-in disk tier for evicted, idle and oversized conversations (--conversation-cache-dir)
open · @AndreyOsipkov · 0 评论 · 去 GitHub 看
Setup & installServer & APIAMD / HIPNVIDIA / CUDAModels & quantsDocumentationWindowsLinux
说明
## Problem and result On a host where the model and its experts leave little RAM, the conversation cache keeps only a few parked conversations. When an agent alternates several long conversations, the oldest one is evicted, and a conversation whose snapshot is larger than the RAM budget is not parked at all. Coming back to either reads the whole prompt again: 18-21 s for 30K tokens here, about 100 s for 200K tokens. With `--conversation-cache-dir` set, such a conversation goes to disk instead of being dropped and is read back on its next turn: 2.0-2.1 s instead of 19-21 s for 30K tokens (table below). Without the flag nothing changes. ## Related discussion #3891 asks for this behaviour (automatic, opt-in, size-capped, no client calls). It overlaps the open #1937, #3341, #4226, #5211, #5399 and #5500, and the seam proposed in #5253; none has a maintainer decision yet. I am opening this because it runs daily on a two-card layer split with little RAM, a setup the others do not report, and two parts are not in them as far as I can see: a snapshot larger than the RAM budget goes straight to disk, and reading one back uses its own 512 MiB RAM floor instead of the parking floor. If a maintainer picks another design, I will close this and port those two parts onto it. ## Approach - **Opt-in.** Four new flags; the disk tier exists only when `--conversation-cache-dir` is set: `--conversation-cache-dir DIR`, `--conversation-cache-disk-gib N` (budget, LRU, at most 256 entries), `--conversation-cache-disk-idle-s S` (an idle parked conversation is written after S seconds), `--conversation-cache-disk-min-gain N` (read from disk only when it reuses at least N more tokens than RAM offers). - **When it writes.** A parked conversation evicted from RAM (`drop_oldest()` calls a hook; a write is skipped when the idle writer has already stored it), an idle one, and a snapshot that does not fit the RAM budget at all (`direct`: written and freed, no RAM floor for it, parked conversations are not evicted for it). - **When it reads.** On a request whose deepest matching resume point is on disk and reuses more than RAM can: the index stores each resume point's length, the hash of its token ids and its picture keys, so a match is found without reading stage files; the loaded tokens are compared again before use. - **Files.** One file per stage plus an index under a folder keyed by model, run configuration and layer layout, so a changed layout starts empty instead of loading something incompatible. A new copy is committed before older ones are dropped; a conversation this run cannot restore is marked and kept, not deleted. Restores survive engine restarts. - **Server.** `/v1/status` gains `disk_entries`, `disk_bytes`, `disk_writes`, `disk_reads` from the engine's `conversation disk:` log lines (0 when the flag is not set). - Design notes and limits: `docs/CONVERSATION_CACHE_DISK.md`. ## Validation Revision `39daf8fd` (3 commits on main `c1ff07f1`). Linux Mint 22.3, CUDA 13.1, driver 595.84, RTX 5060 Ti 16 GB + RTX 3060 12 GB (layer split 17), Core i5-14400F, 62.6 GB RAM, Samsung 990 PRO. **Passed:** - `cmake -DSTRATA_ENABLE_CUDA=OFF -DSTRATA_BUILD_TESTS=ON -DSTRATA_BUILD_CONVERSATION_TESTS=ON` and `ctest -R "conversation|conv_cache"`: 7/7, including the new `conversation_disk_test` (matching at the live end, at a checkpoint and not at all; stage images back in order; a newer copy supersedes the older one; another layout gets its own folder; stray files and a damaged entry after a crash; LRU within the budget; a failed entry kept but not offered; the eviction hook). - `python -m unittest serve.test_monitor`: 11 tests OK (the new disk counters included). - CUDA build of `strata` and `strata_kq`, Release, `CMAKE_CUDA_ARCHITECTURES=86;120`: no warnings in the new files. - Hardware run, Flash-Next IQ3_S, `--kv int8`, MTP, RAM cache cut to 256 MiB / 1 slot so every switch goes through the disk; two conversations of ~30K tokens alternate: | Turn | Prompt tokens read | From cache | Wall | | --- | ---: | ---: | ---: | | A1 | 29,774 | 0 | 20.9 s | | B1 | 27,438 | 0 | 19.3 s | | A2 | 27 | 29,778 | 2.0 s | | B2 | 27 | 27,442 | 2.1 s | Log: `conversation disk: wrote 29778 tokens (direct) in 342.2 ms, 2.37 GB/s`, `conversation disk: read 29778 tokens (29778 reused, live) in 695.3 ms, 1.17 GB/s`; snapshots 740-890 MiB. - The same patches on engine 0.1.45.3 have run daily on this machine since late October 2026: switching between 5 conversations of ~21K tokens 18-19 s -> 3.5-4.5 s per switch, decode unchanged. **Failed:** none observed. **Not run:** - HIP, SYCL and Metal builds (`cmake/strata_gpu_engine.cmake` adds one source file for CUDA and HIP). - Windows (the file layer has a Windows branch; neither built nor run there). - A byte-for-byte comparison of the default path against main. The new code runs only behind `--conversation-cache-dir`; with it unset the only visible change is the four `disk_*` counters in `/v1/status`. - Batch mode (`"parallel": N`) with the disk tier. ## Limits - Whole conversations are written; one that grows by a few hundred tokens per turn is rewritten when it moves again (`--conversation-cache-park-gain` keeps that rare). - Disk use is capped by `--conversation-cache-disk-gib`; a conversation of 200K tokens is 4-5 GB on this model. - A changed model, run configuration or layer layout starts a new folder; old folders are reported at start, not deleted. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
本站相关内容
相关页面的快捷入口。