#6439 [Engine] Persist the SSD conversation tier across restarts when the model fingerprint matches
closed · @mnott · 0 comentarios · En GitHub
Server & APIAMD / HIPNVIDIA / CUDAModels & quantsWindows
Descripción
## Problem and result
The SSD conversation tier (`--conversation-spill-dir`, on by default beside the data pack) parks
spilled conversations as session files, but its folder belongs to one engine process: it is locked
per PID and removed when the engine exits — including on a restart or a reboot. Every long-lived
conversation then pays full prompt re-read cost on the first turn after the restart, even though the
model and settings are unchanged and the parked state on disk was still exactly resumable.
With `--conversation-spill-persist` the tier lives in a stable folder named by the model's
fingerprint — `<root>/fp-<16 hex>` — and its entries survive the process. At start the engine takes
the folder (a lock file; a second live engine of the same model falls back to its own per-process
folder as before), reads each entry's small index file, and adopts every entry whose recorded
identity (model fingerprint + config fingerprint) matches this run: no session-file read is needed
to index them. Entries of another fingerprint, half-written files and orphans are deleted, and the
folders of other fingerprints go with it. Before: everything lost at exit. After: a restart finds
the tier where it left it.
## Related discussion
none
## Approach
- Each persistent entry gets a small index file beside its session file (`conv-N.meta`: identities,
byte size, the live token/image ids, the checkpoint ids). A restart adopts an entry when its index
reads back, its identity matches and the session file's size equals the recorded one; anything else
in the folder is removed. No time-based cleanup: an entry goes only when the size cap needs room
(least recently used first — a matched-and-read entry refreshes its use) or when a longer spill of
the same conversation replaces it (an entry whose tokens are a prefix of the new one is superseded
and deleted at write time). Reads in persistent mode leave the file in place; `close()` keeps it.
- The fingerprint itself had to be fixed first for this to work live: `--mtp` can name an rt
directory (draft_vocab.bin only) whose drafter the Metal engine opens as the GGUF beside it, and
the old fingerprint listed a pack layout that folder does not have — the fingerprint failed, which
closed the tier. That fix is the first commit here and is open separately as #6305: **this PR
depends on #6305** (rebase onto main once it lands).
- Compatibility: off by default; without `--conversation-spill-persist` the per-process folder
behaves exactly as before. No format change to session files; the `.meta` index is new and only
written in persistent mode. A damaged index drops just its entry, never the run.
## Validation
**Passed:**
- `g++ -std=c++20 -I engine/include <test>.cpp engine/src/core/conversation_file.cpp -pthread` + run,
revision fc9110f06 (this branch), macOS arm64, Apple clang:
- `conversation_spill_test`: 96 checks, 0 failed — includes the new persist block: a first run
writes three entries into `fp-<hex>`; a second live engine of the model gets its own folder; a
restart with the same fingerprint adopts all three and reads one back byte for byte; a read
leaves the entry in place; at the cap (`--conversation-spill-mib`) the least recently used entry
goes, not the oldest written; a grown conversation replaces its superseded prefix and entries
that are no prefix stay; a restart with another config or another model fingerprint adopts
nothing and removes the old folder; a cut index file drops only its entry.
- `conversation_file_test`: 209 checks passed (includes the #6305 MTP-resolution cases).
- `conversation_cache_test`: 13534 checks passed.
- Real-world proof: 74 conversations (85794 MiB) adopted after a restart.
**Failed:**
- none observed.
**Not run:**
- CUDA/HIP/SYCL builds and the full engine (`generate.cpp` wiring is host code compiled into the
serve path; only the core tests above were built here). The Windows lock/rename paths of the
stable folder were not exercised on Windows.
## Limits
- Adoption requires an identical model fingerprint: a moved-but-identical model still matches (roles,
not paths), any byte change to a hashed file or a config change drops the tier's files at the next
start by design.
- The cap applies across restarts: if `--conversation-spill-mib` is lowered between runs, adoption
evicts least-recently-used entries down to the new cap.
- Two engines of the same model: only one holds the stable folder; the other spills into its own
per-process folder (not shared, removed at exit).
En el sitio
Enlaces a install, modelos, releases.