#6538 [Engine] SSD conversation tier: fingerprint the resolved MTP files and keep the tier across restarts (v0.2.18)
open · @mnott · 0 commentaires · Sur GitHub
Server & APIAMD / HIPNVIDIA / CUDAModels & quantsWindows
Description
## Problem and result This supersedes #6305 and #6439: the same four commits, rebased onto v0.2.18 (main 6e38f588a), in one PR because the persistence depends on the fingerprint fix. 1. **Fingerprint (was #6305).** `--mtp` can name an rt directory (draft_vocab.bin only) whose drafter the Metal engine opens as the GGUF beside it. The conversation fingerprint listed a pack layout that folder does not have, so the fingerprint failed and the SSD conversation tier turned itself off (issue #6281). Now the fingerprint hashes the MTP files the engine actually resolves (the GGUF itself, or the files of an rt directory). 2. **Persistence (was #6439).** The SSD conversation tier (`--conversation-spill-dir`) belongs to one engine process: its folder is locked per PID and removed at exit, so every long conversation re-reads its whole prompt after a restart. With `--conversation-spill-persist` the tier lives in a stable folder named by the model fingerprint (`<root>/fp-<16 hex>`); at start the engine takes it (lock file; a second engine of the same model falls back to its own per-process folder as before), reads each entry's small index file, adopts the entries whose model and config fingerprint match, and removes everything else. Before: everything lost at exit. After: a restart finds the tier where it left it. 3. A one-line test fix (`serve tests: VisionArgs stubs lifeline's watcher`), needed for `serve/test_server.py` to pass on macOS. ## Related discussion #6281 (the tier turning off with a GGUF/rt drafter), #6305 and #6439 (closed in favour of this PR). ## Approach - Each persistent entry gets an index file beside its session file (`conv-N.meta`: identities, byte size, live token/image ids, checkpoint ids). An entry is adopted when its index reads back, its identity matches and the session file has the recorded size; anything else in the folder is removed. Eviction only by the size cap (least recently used first, a matched-and-read entry refreshes its use) or when a longer spill of the same conversation supersedes it. - Compatibility: off by default; without `--conversation-spill-persist` the per-process folder behaves exactly as before. No format change to session files; the `.meta` index is only written in persistent mode. A damaged index drops just its entry. ## Validation **Passed:** - Core tests, branch `pr/spill-persist-0218` (e157d330b) on v0.2.18, macOS arm64, Apple clang, `g++ -std=c++20 -I engine/include <test>.cpp engine/src/core/conversation_file.cpp -pthread`: `conversation_spill_test` 96 checks, 0 failed (persist block: write, second engine gets its own folder, restart adopts and reads back byte for byte, LRU at the cap, superseded prefixes, other fingerprint adopts nothing, cut index drops one entry); `conversation_file_test` 209 passed (includes the MTP resolution cases); `conversation_cache_test` 13534 passed. - `python3 -m pytest serve/test_server.py -q` on this branch: 445 passed, 685 subtests. - Live, Apple M5 Max 128 GB, Qwen3.8-Flash-Next (Gyro-L, noPLE + PLE Q8_0) with `--mtp <rt dir>`, Metal engine built from this code on v0.2.18: the tier stays on with the rt drafter, and after a restart "SSD tier in …/fp-<hex>, up to 1048576 MiB, kept across restarts: 110 conversations adopted from the last run". Prefix reuse on a 52k-token agent prompt afterwards: 51,120 of 52,011 tokens reused, 891 read in 1.25 s. **Failed:** - none observed. **Not run:** - CUDA/HIP/SYCL builds (Metal and the core tests only). The Windows lock/rename paths of the stable folder were not exercised on Windows. ## Limits - Adoption needs an identical model fingerprint: a moved but identical model still matches (roles, not paths); any byte change to a hashed file or a config change drops the tier's files at the next start, by design. - The cap applies across restarts: lowering `--conversation-spill-mib` between runs evicts least-recently-used entries down to the new cap at adoption. - Two engines of the same model: only one holds the stable folder; the other spills into its own per-process folder.
Sur le site
Liens install, modèles, releases.