#2255 conversation cache: once every slot holds a pinned prefix, nothing is parked any more ("skip parking ... exceeds available budget")

Descripción

**Version:** 0.1.42 (single RTX 3090, Linux, CUDA). `evict_oldest()` is unchanged on `main` (a0116b27, 2026-10-12).

**Setup:** Home Assistant's voice assistant talks to Strata through an OpenAI-compatible gateway that adds `strata_prefix` to every request, so the system prompt + tool list (~6–10K tokens) stay pinned. `--conversation-cache-mib 22528 --conversation-cache-slots 32`, `"parallel": 2`. Other clients (agents, long-context jobs) share the engine.

**What happens:** every Home Assistant conversation carries the pinned checkpoint in its chain, so `SavedConversation::pinned()` is true for all of them. `make_room()` → `evict_oldest()` only evicts entries that are not pinned. Once all 32 slots held pinned conversations, `evict_oldest()` returned false and every attempt to park was refused:

```
strata serve: conversation cache: skip parking (snapshot 451 MiB exceeds available budget)
```

(315 times in the ~2,000 log lines after the last successful park, no park in between, snapshots from 225 to 3,806 MiB alike, while the parked bytes were below the 22 GiB budget and ~11 GiB of RAM were still available — so the slot count with only pinned entries, not the byte budget, refused them). The last successful park before that read `parked=32 ... evictions=309`. Effects: a conversation only resumed from the cache while it was still the active one; everything else was read again — ~4 s instead of ~0.4–0.8 s for a 7K voice request, and three long-context requests (186K–202K tokens) were read from token 0 (~90–98 s each).

**Relation to #1559:** #1559 retires *stale* pins when a new prefix replaces the old one. In our log the pin position changed several times (4578, 5751, 5970, 7250, 6484 tokens — the Home Assistant prompt changes when entities are renamed or exposed), so stale pins probably contributed, and #1559 may already fix part of what we saw; I have not run with #1559 applied. What #1559 cannot cover: a client with a *stable* prefix. Every parked conversation then holds the current pin, #1559 keeps them all pinned, and once `slots` conversations are parked `evict_oldest()` still returns false. This patch is for that case and is independent of #1559.

**Proposed fix (minimal):** when only pinned entries are left, evict the oldest pinned one as long as another holder of a pin remains (the newest is kept), so the shared prefix stays parked and findable. Note that `make_room()` calls `evict_oldest()` in a loop, so a very large incoming snapshot can clear all pinned entries but the newest one, not just one.

```diff
--- a/include/strata/core/conversation_cache.hpp
+++ b/include/strata/core/conversation_cache.hpp
@@
-    // False when none can go (empty, or only entries that hold a pinned shared prefix are left).
+    // False when none can go (empty, or a single entry that holds a pinned shared prefix is left).
     bool evict_oldest() {
         auto victim = std::find_if(entries_.begin(), entries_.end(), [](const SavedConversation& e) { return !e.pinned(); });
-        if (victim == entries_.end()) return false;
+        if (victim == entries_.end()) {
+            // Every parked conversation holds a pinned prefix (a client that pins its system prompt on every
+            // request): keep the newest holder and let the oldest pinned conversation go, otherwise the cache
+            // stays full of pinned entries and nothing is parked any more.
+            const auto holders = std::count_if(entries_.begin(), entries_.end(),
+                                               [](const SavedConversation& e) { return e.pinned(); });
+            if (holders <= 1) return false;
+            victim = std::find_if(entries_.begin(), entries_.end(), [](const SavedConversation& e) { return e.pinned(); });
+        }
         bytes_ -= victim->bytes();
```

(The comment in `make_room()` — "with only pinned ones left the new image does not fit" — would need the same update.)

Test added to `src/core/conversation_cache_test.cpp` (fails on 0.1.42 at "a fourth pinned one still parks", passes with the change; the existing pinned-prefix checks — a single pinned entry is never evicted — still pass, 4,219 checks):

```cpp
    {   // every slot holds the pinned prefix: a new one evicts the oldest pinned conversation,
        // the newest holder keeps the prefix
        auto pin = [&](std::initializer_list<int32_t> ids) {
            auto c = image(ids);
            c.checkpoints.push_back(ConversationCheckpoint{});
            c.checkpoints.back().ids = {1,2};
            c.checkpoints.back().pinned = true;
            c.checkpoints.back().gdn.resize(64, 1);
            c.checkpoints.push_back(ConversationCheckpoint{});   // its own deeper checkpoint, like a real turn
            c.checkpoints.back().ids = ids;
            c.checkpoints.back().gdn.resize(64, 2);
            return c;
        };
        const size_t pb = pin({1,2,3}).bytes();
        ConversationCache full(pb * 4, 3);
        check(full.put(pin({1,2,3})) && full.put(pin({1,2,4})) && full.put(pin({1,2,5})) && full.size() == 3,
              "three pinned conversations fill the slots");
        check(full.put(pin({1,2,6})) && full.size() == 3 && full.evictions() == 1,
              "a fourth pinned one still parks: the oldest pinned conversation leaves");
        check(full.best(std::vector<int32_t>{1,2,3}, {}, true).tokens == 2, "the oldest is gone, the pinned prefix is still found");
        check(full.best(std::vector<int32_t>{1,2,6,9}, {}, true).tokens == 3, "the newest pinned conversation is found whole");
        check(full.put(image({7,8,9})) && full.size() == 3 && full.evictions() == 2, "an unpinned one also gets a slot");
    }
```

**In use since the change** (same machine and clients, ~35 min incl. a 50-request Home Assistant benchmark and a parallel 24K–56K agent conversation): no `skip parking` line (0 vs. 315 before); `parked=32`, `evictions=36`; 54 of 57 Home Assistant requests resumed ~10,459 of ~10,480 tokens from the cache. Short observation window, and the log does not mark which evictions were pinned ones.

Only tested with CUDA on Linux; the change is header-only and backend-independent. Happy to open a PR if that is preferred.