Pull requests / #2719

#2719 --prompt-cache-spread (opt-in): the first checkpoint in every --prompt-cache-every tokens of the conversation stays, so a request that departs far back reads at most that much again

open · @bulhenvaster · 0 comments · View on GitHub

AMD / HIPNVIDIA / CUDAModels & quantsLinux

Description

An agent session grows by short turns and every request saves a checkpoint at its new turn, so least recent use keeps the root and the last few turns - all at the conversation's end.  A request that departs further back (the client's summarization call over the older part, an old tool result rewritten) then falls to the root and reads everything after it again.  With the option, eviction_victim_spread (conv_cache.hpp) keeps the first checkpoint in every --prompt-cache-every tokens as a milestone; the rest rotate by least recent use, the two most recently used always stay, and when only milestones are left the one whose neighbours are closest goes.  Without the option, with fewer than 4 checkpoints or --prompt-cache-every 0 the policy is the plain one.  conv_cache_test covers the new policy.

RTX 5060 Ti (PCIe 3.0 x8) + RTX 3060, --peer-device, IQ3_XXS, --prompt-cache 14: a conversation of 80 turns to 121K tokens, then a request with its first 55 turns and a new instruction (86,182 tokens): without the option 5,023 reused + 81,159 read in 60.3 s, with it 83,387 reused + 2,795 read in 4.1 s.  A day of real agent use (31 automatic compactions): the summarization call read a median 8,062 tokens in 7.2 s instead of ~72K in ~55 s.  The answers can differ in wording (a restored checkpoint and a batched re-read round differently).  Not measured: the RAM of 14 checkpoints at its peak; HIP, SYCL and Metal (the change is host code shared by all of them).

**How it was checked.** Built on its own on top of `main` (296b59d2): CUDA 13.3, sm_86 + sm_120, Linux (Ubuntu), the `strata` target. `conv_cache_test` (header-only) compiled with g++ and passes, including the two new spread cases. The option has run in daily agent use on the rig above since 2026-10-13. HIP, SYCL and Metal were not built: I have only the NVIDIA rig.

One change per PR; nothing else from my fork is in it.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Related on this site

Quick links to related pages.