Pull requests / #65
#65 cache: preserve root system prompt checkpoint across turns and eviction
closed · @code-martin · 0 comments · View on GitHub
Description
### Summary
This PR addresses root checkpoint retention for --serve prompt caching, keeping the root system prompt cached across branching chats and new conversations without compromising v0.1.13 prefill throughput:
1. **Root & Last Turn Checkpoints Only**:
- Rather than checkpointing every turn (which triggers repeated borrow/refill cycles and large memory copies), the prompt reading loop identifies at most two turn boundaries:
- urn_first: the first <|im_start|> past token 0 (the root boundary after the system prompt).
- urn_last: the final <|im_start|> before the assistant turn.
- Long conversation histories remain batched in full chunks (up to 8192 tokens), preserving v0.1.13 speedup.
2. **Root Preservation During Eviction**:
- In checkpoint_at(), when checks.size() > o.prompt_cache, if checks.size() > 2 && checks[0].ids.size() <= checks[1].ids.size(), eviction drops checks[1] (the oldest intermediate turn) instead of checks[0].
- The root system prompt checkpoint remains resident in checks[0]. Subsequent requests sharing the same system prompt match checks[0] and resume immediately (
esume == checks[0].ids.size()) without prefilling the system prompt again.
Related on strata.com
Editorial links to help you install, pick models, or read release notes — not part of the upstream thread.