#6288 [Engine] conversation cache SSD tier: a layer split's conversations and a kept folder across restarts (--conversation-spill-persist on)

open · @xrip · 0 comments · View on GitHub

Server & APIMulti-GPUAMD / HIPNVIDIA / CUDAModels & quantsWindows

Description

## Problem and result

The SSD tier of the conversation cache (`conversation_spill.hpp`, P2 of #5697) has two limits:

- **It does nothing on a layer split.** `spill_ready` refuses an image with stage parts, and the read path and the writer's take-back are gated on `stages.empty()`. On two cards, a conversation the RAM tier evicts is still dropped, and the next turn reads the whole prompt again.
- **Every file goes with the engine.** That is right as a default, but a server that is restarted for an update, a settings change or a reboot loses every parked conversation.

This PR adds both:

- **A layer split:** an entry is one session file per stage.
- **`--conversation-spill-persist on`:** the tier keeps its conversations for the next start of the same model, engine version and settings.

Without the flag, on one GPU, nothing changes: the same per-process folder, the same files and the same file name for the first stage.

Measured on 2x CMP 50HX (sm_75, `--layer-split auto`, layers 0-24 / 25-47), Swift 1.5 IQ3_S, one RAM slot, greedy, `--adapt-swaps 0 --pcie-frac 0.17`:

| Conversation | Cold read | Back from the kept folder after a restart | Same answer |
|---|---:|---:|---|
| A, 41,839 tokens | 21.0 s | 2.7 s | yes |
| B, 41,846 tokens | 21.2 s | 3.0 s | yes |
| L, 165,558 tokens | 66.1 s | 8.4 s | yes |

- **A** went to the disk while the engine ran: the third conversation evicted it, and the background writer wrote it in 2.6 s.
- **B and L** went to the disk at the SIGTERM. The save took 8.8 s: 3 entries, 6 files, 5.7 GB.

## Related discussion

This replaces my PR #1937, a separate disk tier, now that the SSD tier does the same job upstream. The two things #1937 had and the SSD tier lacks are what this PR adds. I close #1937 in favour of this one.

## Approach

**Stages:**
- `conv-N.session` is the first stage's image, and `conv-N.sK.session` the K-th later stage's.
- `put()` writes them in order and removes all of them on a failure. `session_file_write` writes the image it is given, not its `stage_images`, so no copy is made.
- `take_serial()` reads them back into the image and its `stage_images`. A later stage's file is read with the RAM preflight only, because the first stage's read limits name its own layer range. Every stage image is then validated against its runtime by the existing `conversation_snapshot_validate` loop, as for a RAM entry.
- The `stages.empty()` / `stage_images.empty()` gates in `generate.cpp` go.

**The kept folder (`--conversation-spill-persist on`):**
- `SpillNamespace::open(.., kept)` uses `<root>/kept` instead of `<root>/<pid>-<nonce>`. Its `.lock` holds it for one engine, so a second engine is refused with a reason. `close()` leaves the folder on the disk. `clean_dead` never touches it, because the name is not a namespace name.
- Each entry gets an index file, `conv-N.idx`. It holds the session files' identity (model and config fingerprints), the serial, the stage count, the bytes, and the tokens and pictures it matches on. It is written (temp, then rename) after the entry's files and removed before them, so an index always names complete files.
- `open()` takes in the entries whose index has this identity, without reading their files, oldest first within the budget. Everything else in the folder goes: another identity, a damaged index, files no index names, interrupted writes.
- With the flag, the identity is fingerprinted at start instead of at the first spill.
- At QUIT the engine parks the current conversation, evicts every parked one through the evict sink to the background writer, and waits.
- While the folder is kept, the engine ignores SIGINT, SIGTERM and SIGHUP and ends through its stdin. A shutdown sends SIGTERM to every process at once, which would cut the save short.
- The server waits up to 300 s for the engine at QUIT when the flag is on (`quit_s`); otherwise `ENGINE_QUIT_WAIT_S` as before.

The cost rule (`SpillCost::worth`) is unchanged and applies at QUIT too: a conversation cheaper to read again than to round-trip through the disk is not written.

## Validation

Revision 94226066, on 58ee3227 (0.2.16).

**Passed:**
- **Build:** CUDA 13.3, g++-14, `CMAKE_CUDA_ARCHITECTURES=75`, `STRATA_ISA_FLOOR=avx`, targets `strata`, `conversation_spill_test` and `conversation_cache_test`; no errors.
- **`conversation_spill_test`:** 90 checks, 0 failed. The new cases:
  - a split entry round trip, byte for byte, with its files and index removed after the read;
  - the kept folder over three starts: taken in from the index without a read, strays and interrupted writes removed, a new serial after the old ones, another identity's entries removed;
  - a second engine refused the kept folder;
  - the per-process folder still removed at the end.
- **`conversation_cache_test`:** 13,534 checks passed.
- **Server:** `terminal/serve/test_monitor.py` and `test_server_script_mode.py`: 16 passed, 8 subtests passed.
- **End to end on 2x CMP 50HX** (the table above), with the same SSD tier code on the fork, which also carries unrelated kernel patches:
  - the server stopped by SIGTERM;
  - a new server started;
  - the three conversations came back from the kept folder with the same greedy 48-token answers as before the restart.

**Not run:**
- Windows (the lock is `CreateFileW` with no sharing, as before; the signal handling is POSIX-only).
- HIP, SYCL, Metal.
- A crash during the QUIT save: the entries already written stay, because every file is published by rename.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Related on this site

Quick links to related pages.