Pull requests / #3404
#3404 [Engine] conversation cache: drop an older copy of the conversation rewritten turns back (#342 follow-up)
open · @xrip · 0 comments · View on GitHub
Multi-GPUAMD / HIPNVIDIA / CUDAModels & quantsWindows
Description
## Problem and result `ConversationCache::drop_superseded` (#342) drops a parked copy of the outgoing conversation only when the outgoing checkpoint chain holds that copy's deepest checkpoint. That covers a client that re-renders the last reply. It misses a client that rewrites its last turns a few turns back (agents trim tool output and thinking): the copy's deepest checkpoint is past the rewrite, and once the conversation has grown, the chain's spread checkpoints (8 here) no longer hold anything near the copy's end. The stale copy then stays parked: it takes the byte budget (5.7 GB of an 8 GiB budget for a 300K-token copy here) and is evicted later in place of another conversation, or, with a disk tier, written out. - **Before:** on my box one 120K-326K-token agent conversation left two such copies (120,236 tokens, rewritten after token 116,787; 248,844, rewritten after 246,337). At the next switch both were evicted, and with the disk tier of #1937 the park waited 41.4 s for their writes. - **After:** a parked copy that is shorter than the outgoing conversation and has at most a tenth of its tokens not shared with it is dropped as superseded, like the copies #342 drops. ## Related discussion Follow-up to #342 (closed), whose rule this keeps. Found while running the disk tier of #1937; this change does not depend on it. ## Approach - `superseded_at()` (new, private) holds the rule for `drop_superseded()` and `parked_prefix_tokens()`, which had the deepest-checkpoint test twice: first #342's rule unchanged, then the new one (`std::mismatch` for the shared tokens). - Kept, as before: an entry without checkpoints, another conversation that shares only the system prompt with the outgoing one, and an equal or longer entry (a branch the client may go back to). - No new option. Without the conversation cache nothing changes. ## Validation **Passed:** - `conversation_cache_test` (CPU build, `-DSTRATA_BUILD_CONVERSATION_TESTS=ON`, GCC 14 in `nvidia/cuda:13.3.0-devel-ubuntu24.04`): 4,774 checks, among them the new case (a copy rewritten 4 of 40 tokens from its end is dropped, 5 of 41 is kept, with the outgoing chain thinned to the root) and every existing #342, borrowing and park-gain check. `conversation_file_test`: 201 checks. - The rule against the stale copies in my box's log: 3,449 of 120,236, 2,507 of 248,844, 3,049 of 322,930 and 330 of 76,492 tokens not shared, all dropped by it; sub-agents sharing only the ~16K-token system prompt are kept. - Live in my fork (on 0.2.1, 2x CMP 50HX, Ubuntu 24.04.4, Flash-Next IQ3_XXS, conversation cache 8 GiB / 16 slots) since 2026-10-20 12:35: a conversation rewritten at 177,146 tokens dropped its previous copy, and that park took 4.1 s with nothing evicted. The log line does not say which of the two rules dropped it. **Failed:** none. **Not run:** - A live A/B of the two rules over a full day; I will add the counts here. - HIP, SYCL, Metal and Windows builds (a header-only change with no platform code). ## Limits - The tenth is a judgment: a client that goes back to a dropped copy reads again from the nearest checkpoint still held, against the budget the copy holds while nobody comes back to it. - One box and one agent client (Claude Code) measured. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
Related on this site
Quick links to related pages.