#5835 [Engine] Conversation cache: an opt-in disk tier for evicted, idle and oversized conversations (--conversation-cache-dir)

open · @AndreyOsipkov · 0 コメント · GitHub で見る

Setup & installServer & APIAMD / HIPNVIDIA / CUDAModels & quantsDocumentationWindowsLinux

本文

## Problem and result

On a host where the model and its experts leave little RAM, the conversation cache keeps only a few parked conversations.
When an agent alternates several long conversations, the oldest one is evicted, and a conversation whose snapshot is
larger than the RAM budget is not parked at all. Coming back to either reads the whole prompt again: 18-21 s for 30K
tokens here, about 100 s for 200K tokens.

With `--conversation-cache-dir` set, such a conversation goes to disk instead of being dropped and is read back on its
next turn: 2.0-2.1 s instead of 19-21 s for 30K tokens (table below). Without the flag nothing changes.

## Related discussion

#3891 asks for this behaviour (automatic, opt-in, size-capped, no client calls). It overlaps the open #1937, #3341,
#4226, #5211, #5399 and #5500, and the seam proposed in #5253; none has a maintainer decision yet. I am opening this
because it runs daily on a two-card layer split with little RAM, a setup the others do not report, and two parts are
not in them as far as I can see: a snapshot larger than the RAM budget goes straight to disk, and reading one back
uses its own 512 MiB RAM floor instead of the parking floor. If a maintainer picks another design, I will close this
and port those two parts onto it.

## Approach

- **Opt-in.** Four new flags; the disk tier exists only when `--conversation-cache-dir` is set:
  `--conversation-cache-dir DIR`, `--conversation-cache-disk-gib N` (budget, LRU, at most 256 entries),
  `--conversation-cache-disk-idle-s S` (an idle parked conversation is written after S seconds),
  `--conversation-cache-disk-min-gain N` (read from disk only when it reuses at least N more tokens than RAM offers).
- **When it writes.** A parked conversation evicted from RAM (`drop_oldest()` calls a hook; a write is skipped when
  the idle writer has already stored it), an idle one, and a snapshot that does not fit the RAM budget at all
  (`direct`: written and freed, no RAM floor for it, parked conversations are not evicted for it).
- **When it reads.** On a request whose deepest matching resume point is on disk and reuses more than RAM can: the
  index stores each resume point's length, the hash of its token ids and its picture keys, so a match is found without
  reading stage files; the loaded tokens are compared again before use.
- **Files.** One file per stage plus an index under a folder keyed by model, run configuration and layer layout, so a
  changed layout starts empty instead of loading something incompatible. A new copy is committed before older ones are
  dropped; a conversation this run cannot restore is marked and kept, not deleted. Restores survive engine restarts.
- **Server.** `/v1/status` gains `disk_entries`, `disk_bytes`, `disk_writes`, `disk_reads` from the engine's
  `conversation disk:` log lines (0 when the flag is not set).
- Design notes and limits: `docs/CONVERSATION_CACHE_DISK.md`.

## Validation

Revision `39daf8fd` (3 commits on main `c1ff07f1`). Linux Mint 22.3, CUDA 13.1, driver 595.84,
RTX 5060 Ti 16 GB + RTX 3060 12 GB (layer split 17), Core i5-14400F, 62.6 GB RAM, Samsung 990 PRO.

**Passed:**
- `cmake -DSTRATA_ENABLE_CUDA=OFF -DSTRATA_BUILD_TESTS=ON -DSTRATA_BUILD_CONVERSATION_TESTS=ON` and
  `ctest -R "conversation|conv_cache"`: 7/7, including the new `conversation_disk_test` (matching at the live end, at a
  checkpoint and not at all; stage images back in order; a newer copy supersedes the older one; another layout gets
  its own folder; stray files and a damaged entry after a crash; LRU within the budget; a failed entry kept but not
  offered; the eviction hook).
- `python -m unittest serve.test_monitor`: 11 tests OK (the new disk counters included).
- CUDA build of `strata` and `strata_kq`, Release, `CMAKE_CUDA_ARCHITECTURES=86;120`: no warnings in the new files.
- Hardware run, Flash-Next IQ3_S, `--kv int8`, MTP, RAM cache cut to 256 MiB / 1 slot so every switch goes through the
  disk; two conversations of ~30K tokens alternate:

  | Turn | Prompt tokens read | From cache | Wall |
  | --- | ---: | ---: | ---: |
  | A1 | 29,774 | 0 | 20.9 s |
  | B1 | 27,438 | 0 | 19.3 s |
  | A2 | 27 | 29,778 | 2.0 s |
  | B2 | 27 | 27,442 | 2.1 s |

  Log: `conversation disk: wrote 29778 tokens (direct) in 342.2 ms, 2.37 GB/s`,
  `conversation disk: read 29778 tokens (29778 reused, live) in 695.3 ms, 1.17 GB/s`; snapshots 740-890 MiB.
- The same patches on engine 0.1.45.3 have run daily on this machine since late October 2026: switching between 5
  conversations of ~21K tokens 18-19 s -> 3.5-4.5 s per switch, decode unchanged.

**Failed:** none observed.

**Not run:**
- HIP, SYCL and Metal builds (`cmake/strata_gpu_engine.cmake` adds one source file for CUDA and HIP).
- Windows (the file layer has a Windows branch; neither built nor run there).
- A byte-for-byte comparison of the default path against main. The new code runs only behind
  `--conversation-cache-dir`; with it unset the only visible change is the four `disk_*` counters in `/v1/status`.
- Batch mode (`"parallel": N`) with the disk tier.

## Limits

- Whole conversations are written; one that grows by a few hundred tokens per turn is rewritten when it moves again
  (`--conversation-cache-park-gain` keeps that rare).
- Disk use is capped by `--conversation-cache-disk-gib`; a conversation of 200K tokens is 4-5 GB on this model.
- A changed model, run configuration or layer layout starts a new folder; old folders are reported at start, not deleted.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

関連リンク

インストール・モデル・リリースへのサイト内リンク。