#3957 [Engine] serve: a long conversation survives an engine restart - parked conversations saved at a graceful stop and loaded at the next start, to files or (Windows) kept in RAM with no disk writes (opt-in STRATA_PARK_DIR, STRATA_PARK_RAMLIKE=1; layer split included)
open · @ruibeikaa · 0 评论 · 去 GitHub 看
Server & APIMulti-GPUAMD / HIPNVIDIA / CUDAModels & quantsSecurityDocumentationWindowsLinux
说明
## What this improves
- **The next turn of a long conversation after a restart takes seconds, not minutes.** Today every stop of the engine forgets the conversation cache, so the first turn after a restart reads the whole conversation again from token 0. With `STRATA_PARK_DIR=<folder>` the parked conversations are saved at a graceful stop and put back in the cache at the next start, before the first request. Measured on four GP100 dies (Swift 1.5 GGUF, `--layer-split 12,24,36`): a 260,834-token conversation that takes **296 s** to read from zero was saved in 3.6 s and loaded in 3.2 s, and its next turn after the restart took **7.2 s** (260,834 tokens reused, 893 read). The saving grows with the conversation: a re-read costs roughly its length divided by the prompt speed, a load costs its size divided by the disk (or memory) speed.
- **A knob for no disk writes at all (Windows): `STRATA_PARK_RAMLIKE=1`.** Saving to files writes several GB per long conversation at every graceful stop, which spends an SSD's write endurance. With the knob, the same session bytes go to named memory sections that a small keeper process holds in RAM from the stop to the next start. For a 103,749-token conversation the stop wrote **3.4 MiB** to the system disk **instead of 1.94 GB**, no page went to the pagefile in the ten minutes after it, and the next turn after the restart reused 103,721 tokens and took **975 ms** instead of 94-95 s from zero, with the same reply.
- **It is what real agent sessions lose today.** In six days of one coding-agent client's logs on that rig, five restarts of the engine came with a long conversation open, and each first turn read it again from zero: 33K-382K tokens, **40-378 s each, about 1,180 s in all (~200 s a day)**, and the user waits that long on the first message after each. With the engine file and its switches unchanged, this brings such a restart back in seconds; one of the five crossed an engine update, which it deliberately does not load (below).
- **Layer splits too.** Session files (`SAVE` / `RESTORE`) refuse `--layer-split` (#3539, #3744); here each stage of a conversation is its own session image, so multi-GPU splits keep their conversations. A single GPU takes the same path.
- **Who benefits:** anyone who stops and starts the engine while long conversations are open - for the night, to free the GPUs for another program and back, the idle unload, `POST /unload`, or a switch to another model and back within `STRATA_PARK_MAX_AGE_H` - and most of all long agent sessions (100K-400K tokens) on slower prompt reads. The RAM knob is for Windows users who have the RAM to spare between the stop and the next start and do not want the disk writes.
What it costs (all opt-in; unset, nothing is written, kept or read):
- **Files (default mode):** disk writes at every graceful stop, for the conversations that changed since the last save (5.33 GB for the 260K one). To keep them small a save skips conversations already in the folder and not run since (a stop with no request since the last load writes 0 bytes), conversations under `STRATA_PARK_MIN_TOKENS` (default 16384) and ones parked more than `STRATA_PARK_MAX_AGE_H` ago (default 48 h), and writes only two checkpoints of each. Disk space up to the conversation cache's budget, and a longer stop by the save's time (3.6 s for 5.33 GB on an NVMe SSD).
- **RAM knob:** RAM and commit equal to the kept conversations' size (1.94 GB for 103K tokens; at most the conversation cache's budget) from the stop until the next start loads them, or until `STRATA_PARK_MAX_AGE_H` (the keeper then ends by itself). A save is refused when less than `STRATA_PARK_RAMLIKE_FLOOR_MIB` (8192) of commit would be left. Every stop copies every kept conversation again (RAM to RAM, ~1.4 s for 1.94 GB). Lost at a reboot or sign-out, or when the keeper cannot outlive the server (a job that ends its processes and allows no breakaway); then the next start reads the conversations again, as without the change, and says why. Windows only.
- (My own setting is the RAM knob with `STRATA_PARK_MIN_TOKENS=50000` and `STRATA_PARK_MAX_AGE_H=1`: long sessions only, restarts within the hour.)
- **Both:** graceful stops only (a kill, a crash or a power cut keeps nothing new; with files the previous save stays). The same engine and settings only: the saved images are bound to the model, the engine file (size and modification time) and every `STRATA_*` variable except `STRATA_PARK_*` (plus the session files' own config fingerprint); after an update or a changed switch nothing is loaded and the next stop replaces them.
## Problem and result
A restart of the engine drops every parked conversation, and the next turn of a long one reads its whole prompt again. The session files keep one conversation, need a client to save and restore it, and refuse a layer split.
| | before | `STRATA_PARK_DIR` (files) | `+ STRATA_PARK_RAMLIKE=1` (RAM, Windows) |
|---|---|---|---|
| first turn after a graceful restart | read from zero (260,834 tokens: 296 s) | loaded at start (3.2 s), turn 7.2 s, 260,834 reused | 103,749 tokens: loaded in 3.0 s, turn 975 ms, 103,721 reused (from zero: 94 s) |
| written to the disk at the stop | - | the changed conversations (5.33 GB for 260K; 0 bytes after a stop with no request) | 3.4 MiB (two small files and logs); 0 pagefile writes in the next 10 min |
| reply after the restart | - | bit for bit the one without a restart (30K conversation, two restarts) | bit for bit the one without a restart |
| `--layer-split` | SAVE / RESTORE refused | one session image per stage | one section per stage |
| unset (default) | - | nothing written or read; the INFO line has one more key | - |
Why named sections and not a file marked temporary: on Windows 11 a file created with `FILE_ATTRIBUTE_TEMPORARY` (also with `FILE_FLAG_DELETE_ON_CLOSE`) and never flushed was written to the disk as it was written - 4 GiB of 4 GiB, with the handle still open (1 GiB of 1 GiB with `DELETE_ON_CLOSE`). A named, pagefile-backed section (`CreateFileMappingW(INVALID_HANDLE_VALUE, ..., SEC_COMMIT)`) is not file cache: its pages stay in the working set of a process that maps it and touches them, and they go to the pagefile only if Windows trims that working set. The sections live as long as one process holds them, so a keeper outside the server's process tree holds them while no engine runs.
## Related discussion
- #1090: the maintainers said one disk-persistence design will be picked among the open ones. This is a narrower one than the disk tiers: nothing changes while the engine runs (no eviction to disk, no read path in requests); it only saves the parked list at a graceful stop and loads it at the next start, through the existing session-file format and identity checks - to files, or with the knob to memory sections that never touch the disk.
- #1937 (open) also persists conversations across restarts (`--conversation-disk-persist on|exit-only`, one session file per stage) as part of a disk tier for evictions. The difference here: no change to the cache's eviction or request path, a save writes only what changed since the last one (and nothing when nothing ran), short and stale conversations are not written, two checkpoints per conversation instead of every one, the RAM knob with no disk writes, and it was run on Windows with a four-stage split. If the maintainers prefer #1937's design, the write-reduction rules and the RAM knob can go on top of its `exit-only` path instead.
- #3341 and #3565 park through session files during a run (a disk budget; switches between conversations); #3539 and #3744 ask for session files under `--layer-split`, which this covers for the restart case only.
- No scope issue was opened: it is opt-in and changes no default.
## Approach
Three commits: the save / load at stop and start (files), the RAM knob on top, then a bound on the hand-over to the keeper.
- `src/program/generate.cpp`: two commands between requests, `PARKSAVE <folder>` and `PARKLOAD <folder>`, answered like `SAVE` / `RESTORE` (`SAVED` / `RESTORED <conversations> <bytes> <ms>`, `SESSION` / `SWAIT` on the way, `SERR` for a refusal that leaves the engine in step).
- PARKSAVE parks the live conversation, waits for the background copies (`STRATA_CKPT_ASYNC`, `STRATA_PARK_ASYNC`), as `SAVE` does, then writes every parked conversation that is kept. A parked image is one plain single-carve `SavedConversation` per stage (the park already moved the checkpoints' stage parts apart), so each stage goes through `session_file_write` as its own v1 image under the same model/config identity as `SAVE`. The image lends only the two checkpoints it writes (the deepest, where the next turn resumes, and the shallowest, the system prompt's root) for the length of one write and gets the whole chain back, also on a failure.
- Files are named after a hash of the conversation's token ids and its length; one already in the folder whose conversation has not run since is not written again (a conversation that runs again is taken out and parked as a new image). A manifest (`park.manifest`, format 2: the tag, the stage count, one line per conversation with its parked time) is written to a temporary name and renamed over the old one last; only then are the files it no longer lists removed. A save that fails keeps the old manifest and every file it lists.
- The tag: a hash of `STRATA_VERSION`, the engine file's size and modification time, the sorted `STRATA_*` environment (not `STRATA_PARK_*`) and the stage count. Another tag loads nothing.
- PARKLOAD reads each listed conversation's stage images (the same RAM admission, read limits and `conversation_snapshot_validate` as `RESTORE`, before anything is put back) and `put`s the image in the parked list; the next matching request restores it the usual way. A conversation that does not validate or fit is skipped with a log line; the others still load.
- Refused with `--peer-device`, `--batch` and without the conversation cache.
- `SavedConversation` gets two run-time fields (`parked_unix`, `disk_name`; not in the file) and `ConversationCache` read access to its entries; the park stamps `parked_unix`.
- The INFO line says `park_files=1`, or `park_files=2` on Windows builds (files and sections), so the server knows what the engine takes.
- The RAM knob, `STRATA_PARK_RAMLIKE=1` (Windows only; elsewhere a `"mem:"` path and the variable are refused with a message):
- `src/core/conversation_file.{hpp,cpp}`: `session_file_write` / `session_file_read` take `"mem:<name>"` for a named section holding the session file's exact bytes behind a 64-byte header (`[u64 length]`). A write has no temporary file, no rename and no flush; its preflight is the free commit (a `SEC_COMMIT` section charges its whole size when it is created) instead of the free disk space; an existing name is refused, never written into. A written section stays held (handle and view) until the process exits, so a keeper can open it by name meanwhile. `session_section_exists` for the name check.
- `generate.cpp`: each kept conversation's stage images go to `mem:Local\strata-park-<name>-s<stage>` (the name also carries the engine's process id and the second, so a name is never reused), the manifest is `park-mem.manifest`, every kept conversation is written anew at each save, and no `.ses` files are removed.
- `serve/park_keeper.py`: after PARKSAVE the server starts the keeper through a middle process that exits at once (so the keeper is nobody's descendant and a tree kill of the server passes it by; out of the server's job when the job allows a breakaway). The keeper opens every section the manifest lists, maps it, touches its pages for at most 8 s (the rest of the walk goes on afterwards, and every page again every 30 s), sets a soft working-set floor and writes `keeper.json`; the server waits for that before it closes the engine, at most `STRATA_PARK_KEEPER_WAIT_S` (default 30 s) in all. A hand-ov本站相关内容
相关页面的快捷入口。