Pull requests / #2077

#2077 serve: --prompt-cache-file <path>, the long prompt prefix saved to disk and restored across restarts (opt-in)

open · @imanu86 · 0 コメント · GitHub で見る

Server & APIMulti-GPUNVIDIA / CUDAModels & quantsWindows

本文

## Title
Issue: none (an opt-in feature: the prompt prefix survives a restart).

## Summary
`--prompt-cache-file <path>` (opt-in) writes the root checkpoint of the shared prompt prefix to a file and reads it
back at the next start, so the first turn after a restart resumes from the saved prefix instead of reading it again.
Format v2 holds one image per `--layer-split` block (so it works with two cards) and the MTP drafter's root cells.
Without the option nothing changes.

## Why
The in-RAM prompt cache (`--prompt-cache-tail` / `--prompt-cache-every`) dies with the engine. An agent session
(Claude Code through a local gateway here) starts with a 40-130K prefix that is the same across restarts; every
restart of the server re-read it. Measured on this box the first 40K prompt after a restart went from **~42 s** to
**16-20 s** with the file.

## What changed
- `src/core/prefix_cache_file.cpp`, `include/strata/core/prefix_cache_file.hpp` (new, library `strata_prefix_cache`):
  the file format (header with the engine's identity, one image per block, the drafter's root), write and read,
  BCrypt SHA-256 on Windows / the C++ fallback elsewhere.
- `src/program/generate.cpp`: the option; the save after the root checkpoint is taken, the restore at start-up
  before the first request; host memory checked with `available_host_bytes()` before the restore; the file's
  **identity** covers the executable, the arguments and the `STRATA_*` / `CUDA_*` environment, so a file written by
  another configuration is replaced, never misread.
- `src/core/mtp.cpp`, `include/strata/core/mtp.hpp`: the drafter uses `first_needed()` so every cell of the saved root
  is computed, not only the ones the current window asks for.
- `src/core/prefix_cache_file_test.cpp`: the format, round trip and identity checks.
- `CMakeLists.txt`: the library and the test.

## Relation to open work
#1937 (parked conversations on the disk, `--conversation-disk-dir`) and #1810 (conversation checkpoints with disk
restore) persist **conversations** of the conversation cache; this persists the **shared prompt prefix root** (the
`--prompt-cache` checkpoint every conversation of a session starts from, including the drafter's state) and is
independent of the conversation cache. They can coexist; none of them is merged at the time of writing.

## Measurements
RTX 2080 Ti 22 GB + RTX 3060 12 GB, Windows 11, `--layer-split 16 --pipeline-windows 2`, Qwen3.8-Flash-Next IQ3_XXS.
Reading a 40K-token first prompt takes ~42 s on this box; with the file the first turn after a restart resumes from
the saved root instead (one image per block read from an NVMe). Decode and later turns are unchanged; the output tokens
are the same with and without the file (the same checkpoint, read from RAM or from the file). The save/restore path has
run in this fork's daily build since 0.1.40 with 40-130K prefixes. On v0.1.43 the branch was built and started twice
with the option (`prefix disk cache: enabled`, identity computed in 17-23 ms), same answers.

## Risks / limits
Opt-in. The file is rewritten when the identity changes (a new engine build, a changed option): one re-read, as
without the option. A file of a different format version is ignored and replaced.

Part of a series from a public patch registry (`stack/` in the fork imanu86/Strata-2080Ti).

関連リンク

インストール・モデル・リリースへの站内リンク。