Pull requests / #65

#65 cache: preserve root system prompt checkpoint across turns and eviction

closed · @code-martin · 0 comentarios · En GitHub

Server & API

Descripción

### Summary

This PR addresses root checkpoint retention for --serve prompt caching, keeping the root system prompt cached across branching chats and new conversations without compromising v0.1.13 prefill throughput:

1. **Root & Last Turn Checkpoints Only**:
   - Rather than checkpointing every turn (which triggers repeated borrow/refill cycles and large memory copies), the prompt reading loop identifies at most two turn boundaries:
     - 	urn_first: the first <|im_start|> past token 0 (the root boundary after the system prompt).
     - 	urn_last: the final <|im_start|> before the assistant turn.
   - Long conversation histories remain batched in full chunks (up to 8192 tokens), preserving v0.1.13 speedup.

2. **Root Preservation During Eviction**:
   - In checkpoint_at(), when checks.size() > o.prompt_cache, if checks.size() > 2 && checks[0].ids.size() <= checks[1].ids.size(), eviction drops checks[1] (the oldest intermediate turn) instead of checks[0].
   - The root system prompt checkpoint remains resident in checks[0]. Subsequent requests sharing the same system prompt match checks[0] and resume immediately (
esume == checks[0].ids.size()) without prefilling the system prompt again.

En el sitio

Enlaces a install, modelos, releases.