Pull requests / #2915
#2915 conversation cache: park and restore conversations through session files
open · @dsenchankau · 0 comments · View on GitHub
Server & APIAMD / HIPNVIDIA / CUDAModels & quantsDocumentation
Description
## Title
conversation cache: park and restore conversations through session files (`--conversation-cache-disk-mib`)
## Motivation
The idea for this PR was taken from Strata's fork for GLM5.3-Flash named [Maya](https://github.com/mw00/project-maya), where the disk-based conversation cache is enabled by default.
As of now there are only 2 options for the conversation cache in Strata: you either store the cache in RAM or don't use it at all.
RAM cache works fine on smaller quants (e.g. IQ3_S) and large system RAM (e.g. 96GB with the previously mentioned quant). However with a bigger quant like UD-Q4_K_XL and/or smaller system RAM it becomes an issue.
UD-Q4_K_XL's experts are significantly larger, and the OS overhead+model+engine+coding harness after startup already use up (almost) all the RAM. Which forces the user to either offload some of the experts to a disk, sacrificing both decode and prefill performance or skip the RAM cache and deal with constant prompt reprocessing (which takes longer, the bigger the model is).
A much better idea performance-wise is to move the conversation cache to disk and store more of the model's experts in RAM.
It does not really matter how fast or slow the end user's disk is. It will always be faster to write and read the cache from disk than to constantly reprocess the sessions. Especially when actively using subagents and thus switching in-between main and secondary sessions.
As per the repo guidelines, I haven't enabled the disk cache by default, leaving it an opt-in feature instead. But once verified and approved, it can be the new norm. Leaving RAM cache an opt-in feature instead.
## Summary
Opt-in disk-backed conversation cache. `--conversation-cache-disk-mib N` parks conversations to
`conv-*.sess` files under `<pack>/conversation_cache` (budget counts the file bytes), so far more
stay parked than fit in RAM; one returning on a later turn - same run or after a restart - is taken
back from its file and continues from its own state instead of re-reading its prompt. Default off:
the RAM cache and its budget accounting are byte-for-byte the old path.
## What changed
- `include/strata/core/conversation_cache.hpp` - disk mode: `enable_disk`, `put_disk` (budget counts
file bytes), the folder-index slim entry, and the budget-refusal that keeps survivors untouched.
- `src/core/conversation_file.{hpp,cpp}` - meta-only read (`session_file_read_meta`) and the startup
folder index (`conversation_cache_index`) that keeps this model's fresh `conv-*.sess` and removes
foreign/stale/unreadable files; reuses the existing session-file format.
- `src/program/generate.cpp` - the park and restore calls in the serve loop, plus the skip guards
(snapshot over the whole budget, RAM admission).
- `include/strata/core/{conversation_snapshot.hpp,session_save_reclaim.hpp}`, `src/core/conversation_state.cpp`
- the meta/checkpoint plumbing the index and restore need.
- Tests: `conversation_cache_test.cpp` disk-mode budget-refusal check; `conversation_file_test.cpp`
exercises the meta-only read and the folder-index purge.
- `docs/DETAILS.md` - the flag and the measured numbers.
## Extra Notes
Measured on one RTX 4080 SUPER, IQ3_S pack, greedy, `--prompt-cache 4`:
park `1962 tokens to disk (266,114,352 bytes)` -> restart -> `took back 1962 tokens from disk`;
the continuation reads only the new turn (`cached_tokens 1962`, `prompt_n 17`). Guards checked:
a snapshot over the whole budget is refused with an empty folder; foreign and `conv-hacked-9.sess`
files are purged at startup; `--conversation-cache-days 0` empties the folder.
Tests: `ctest` 5/5 pass (`conversation_cache_test` 4789 checks, `conversation_file_test` 217 checks,
plus split_failure / memory / session_save_reclaim); `python -m unittest discover -s serve` 848 OK,
11 skipped.
What I did not test: the change is common code only (no backend sources or `#ifdef`), so it is
backend-agnostic and the unit tests run CPU-only with no GPU - but the park/restore above was
measured only on NVIDIA/CUDA. I did not build or run the HIP or SYCL engine; a maintainer or a
reporter with that hardware should confirm both build and, if convenient, one run on each.
Related on this site
Quick links to related pages.