Issues / #2930

#2930 #342 again on long conversations: the superseded-copy check misses the previous turn's parked copy once its boundary checkpoint rotates out of the `--prompt-cache` chain, so the parent is evicted after `slots` thinking turns

open · @petski3 · 0 comentários · No GitHub

Server & APIMulti-GPULinux

Descrição

Engine 0.1.43.1 (a0116b2), same code in 0.1.44.1 and 0.1.45 (477ac13) (`include/strata/core/conversation_cache.hpp`, `drop_superseded`). Linux, 2x gfx906 layer split, `--conversation-cache-mib 20480 --conversation-cache-slots 4`, `--prompt-cache` at its default 6, `--prompt-cache-every` 16384. Client: pi (OpenAI chat completions, reasoning on, reasoning not sent back in the history).

Workflow: MAIN -> SUB (many turns) -> MAIN, as in #342. MAIN is evicted again once SUB's conversation is long.

What the log shows (one SUB conversation, MAIN parked):

```
prompt 82089 tokens = 81686 reused + 403 read ..., 2226 generated ..., 6 checkpoints
conversation cache: parked 84314 tokens in 21658.4 ms; parked=4 bytes=8650931864 evictions=37 ...
prompt 86065 tokens = 82084 reused + 3981 read ...          <- resumed at the turn boundary, 5 tokens before the previous prompt's end
prompt 88102 tokens = 88065 reused + 37 read ...            <- 5 continuation turns, each extends the live conversation
prompt 88491 tokens = 88454 reused + 37 read ...
prompt 89260 tokens = 89238 reused + 22 read ...
prompt 92558 tokens = 89497 reused + 3061 read ..., 4607 generated
conversation cache: parked 97165 tokens in 10264.4 ms; parked=4 bytes=9217013552 evictions=38 ...   <- no "dropped superseded copy" line
...
conversation cache: parked 191846 tokens in 24323.4 ms; parked=4 bytes=12335948288 evictions=41 ...
prompt 82329 tokens = 0 reused + 82329 read in 173952 ms    <- MAIN comes back, re-read from token 0
```

Every turn whose reply contained thinking is re-rendered by the client without it, so the next prompt diverges 5 tokens before the previous prompt's end, the engine resumes at the turn-boundary checkpoint and parks the outgoing state (the #342 case). `drop_superseded` then looks for that copy's deepest checkpoint in the outgoing chain by exact equality. On a long conversation the chain's 6 slots hold the root plus 4-5 periodic (16K) checkpoints, every continuation turn adds a turn-boundary checkpoint, and the LRU rotation drops the boundary the copy was parked at within a few turns. In the stretch above there were 5 continuation turns between the two parks (elsewhere 20+), so the copy is never recognised, `make_room` evicts oldest-first and MAIN goes after 4 such parks. The eviction counter climbs on every park (35 -> 41) with `parked=4` throughout.

Raising `--prompt-cache` does not fix it (20+ turns between parks). `--conversation-cache-min-tokens` does not apply (SUB is longer than MAIN).

Proposed fix: when the outgoing chain holds a checkpoint deeper than the copy's deepest one (the boundaries that pushed it out), the outgoing conversation *starting with* that checkpoint says the same thing as holding it. It continued from there, so the entry is this conversation a turn or more back:

```cpp
        auto held = [&](const ConversationCheckpoint& c) {
            if (c.ids == ids && c.imgs == images) return true;
            for (const auto& k : checkpoints)
                if (k.ids == c.ids && k.imgs == c.imgs) return true;
            // the chain rotated the boundary out: newer boundaries are in it and the ids still start with the old one
            bool deeper = false;
            for (const auto& k : checkpoints) deeper = deeper || k.ids.size() > c.ids.size();
            return deeper && conversation_prefix(c, ids, images) > 0;
        };
```

A sibling conversation that shares only the root (or a branch) has its own, deeper, boundary checkpoint which the outgoing ids do not start with, so it is kept. A chain with nothing deeper than the entry has not rotated anything, so the equality rule stays the whole rule there. All existing cases in `conversation_cache_test.cpp` still pass (including "ties prefer most recently parked" and the "what is NOT superseded" block). A test that reproduces the rotation (three continuation turns per park with a 3-slot chain) fails on the current header (`one parked copy of SUB at a time although its boundary rotated out`) and passes with the change. Running it patched here since 2026-10-14.

Side observation from the same run, maybe worth its own issue: with the mmap arena (`STRATA_ARENA_MMAP=1`) on a 64 GB box the parked snapshots (12.3 GB at the time) were paged out to swap because `--conversation-cache-min-free-mib` reads MemAvailable, which counts the experts.bin page cache as free; parking then took 20-28 s per switch and the watchdog ended the engine on a 60 s stall in "parking the previous conversation". A floor against anonymous RAM (MemAvailable minus the engine's own mapped file pages) would have refused those parks.

No site

Links install, modelos, releases.