Pull requests / #2077
#2077 serve: --prompt-cache-file <path>, the long prompt prefix saved to disk and restored across restarts (opt-in)
open · @imanu86 · 0 Kommentare · Auf GitHub
Server & APIMulti-GPUNVIDIA / CUDAModels & quantsWindows
Beschreibung
## Title Issue: none (an opt-in feature: the prompt prefix survives a restart). ## Summary `--prompt-cache-file <path>` (opt-in) writes the root checkpoint of the shared prompt prefix to a file and reads it back at the next start, so the first turn after a restart resumes from the saved prefix instead of reading it again. Format v2 holds one image per `--layer-split` block (so it works with two cards) and the MTP drafter's root cells. Without the option nothing changes. ## Why The in-RAM prompt cache (`--prompt-cache-tail` / `--prompt-cache-every`) dies with the engine. An agent session (Claude Code through a local gateway here) starts with a 40-130K prefix that is the same across restarts; every restart of the server re-read it. Measured on this box the first 40K prompt after a restart went from **~42 s** to **16-20 s** with the file. ## What changed - `src/core/prefix_cache_file.cpp`, `include/strata/core/prefix_cache_file.hpp` (new, library `strata_prefix_cache`): the file format (header with the engine's identity, one image per block, the drafter's root), write and read, BCrypt SHA-256 on Windows / the C++ fallback elsewhere. - `src/program/generate.cpp`: the option; the save after the root checkpoint is taken, the restore at start-up before the first request; host memory checked with `available_host_bytes()` before the restore; the file's **identity** covers the executable, the arguments and the `STRATA_*` / `CUDA_*` environment, so a file written by another configuration is replaced, never misread. - `src/core/mtp.cpp`, `include/strata/core/mtp.hpp`: the drafter uses `first_needed()` so every cell of the saved root is computed, not only the ones the current window asks for. - `src/core/prefix_cache_file_test.cpp`: the format, round trip and identity checks. - `CMakeLists.txt`: the library and the test. ## Relation to open work #1937 (parked conversations on the disk, `--conversation-disk-dir`) and #1810 (conversation checkpoints with disk restore) persist **conversations** of the conversation cache; this persists the **shared prompt prefix root** (the `--prompt-cache` checkpoint every conversation of a session starts from, including the drafter's state) and is independent of the conversation cache. They can coexist; none of them is merged at the time of writing. ## Measurements RTX 2080 Ti 22 GB + RTX 3060 12 GB, Windows 11, `--layer-split 16 --pipeline-windows 2`, Qwen3.8-Flash-Next IQ3_XXS. Reading a 40K-token first prompt takes ~42 s on this box; with the file the first turn after a restart resumes from the saved root instead (one image per block read from an NVMe). Decode and later turns are unchanged; the output tokens are the same with and without the file (the same checkpoint, read from RAM or from the file). The save/restore path has run in this fork's daily build since 0.1.40 with 40-130K prefixes. On v0.1.43 the branch was built and started twice with the option (`prefix disk cache: enabled`, identity computed in 17-23 ms), same answers. ## Risks / limits Opt-in. The file is rewritten when the identity changes (a new engine build, a changed option): one re-read, as without the option. A file of a different format version is ignored and replaced. Part of a series from a public patch registry (`stack/` in the fork imanu86/Strata-2080Ti).
Mehr auf der Site
Links zu Install, Modellen, Releases.