#2228 Conversation cache falls back to 32,768-spaced checkpoints for a fast client: 13 re-reads of 6.8K-47K tokens (4-17 s TTFT) in 3.5 h while its prompts are pure appends

本文

**Summary.** With a real remote agent client (Windows, OpenAI-style `/v1/chat/completions` through a reverse proxy, screenshots in the conversation), 0.1.43.1 repeatedly restores a **coarse** checkpoint instead of one at the last turn boundary, and re-reads everything since: 13 occurrences of 6.8K-47K fresh tokens (4-17 s TTFT) in 3.5 h, while the client's prompts are **pure appends** (verified: every earlier message is byte-identical between consecutive requests).

**Rig.** RTX 4090D 48 GB, Windows 11, 128 GB RAM, NVMe; engine 0.1.43.1 release build; Qwen3.8-Flash-Next GSQ-RCO IQ3_XXS native pack, all 24,576 experts resident; `--prefill auto:32768 --spec 4 --spec-min-p 0.70 --pcie-frac 0.35 --mtp <rt> --max-context 262144 --kv int8 --kv-resident 32768 --prompt-cache 12 --conversation-cache-mib 32768 --conversation-cache-slots 8 --vision --vram-reserve-mib 700 --embd-gguf` (+ `--mtp-hnorm stream`, measured neutral here). Vision runs on the CPU (`strata-vision.exe`, 16 threads).

**What the log shows.** Every big read is preceded by a restore of a *checkpoint* (not the live state), and the restore points sit on an exact **32,768-token grid**:

```
restored 53533  tokens (checkpoint) ...  prompt  76136 =  53533 reused + 22603 read (7288 ms)
restored 53533  tokens (checkpoint) ...  prompt  91146 =  53533 reused + 37613 read (13921 ms)
restored 86301  tokens (checkpoint) ...  prompt 102085 =  86301 reused + 15784 read (5663 ms)
restored 86301  tokens (checkpoint) ...  prompt 132087 =  86301 reused + 45786 read (14761 ms)
restored 119069 tokens (checkpoint) ...  prompt 160267 = 119069 reused + 41198 read (15609 ms)
restored 151837 tokens (checkpoint) ...  prompt 198829 = 151837 reused + 46992 read (16943 ms)
restored 184605 tokens (checkpoint) ...  prompt 215114 = 184605 reused + 30509 read (11119 ms)
restored 184605 tokens (checkpoint) ...  prompt 222073 = 184605 reused + 37468 read (15608 ms)
restored 217373 tokens (checkpoint) ...  prompt 224223 = 217373 reused +  6850 read (3920 ms)
restored 217373 tokens (checkpoint) ...  prompt 226654 = 217373 reused +  9281 read (5712 ms)
```

53533 + 32768·k = 86301 / 119069 / 151837 / 184605 / 217373 — every restore point is on that grid. Meanwhile the same conversation, a few requests earlier or later, reuses to 224,977 / 227,567 / 228,144 (i.e. per-turn positions), so the deep checkpoints *do* exist at other moments.

**The client is not changing anything.** I captured the raw request bodies (message-level sha1 per message). Consecutive requests of this client are pure appends — the first differing message index always equals the *previous* message count (e.g. 420->422->425->427->430->435), so every earlier message is byte-identical. One request in the middle of that run still fell back to 217,373 (~95.9% depth) and re-read 9,281 tokens (5.7 s).

**What happens right before the fallback** (one example, the 226654 request):

```
prompt 225020 tokens = 224977 reused + 43 read ... 12 checkpoints
conversation cache: dropped 1 superseded copy of this conversation; parked=3
conversation cache: parked 225122 tokens in 684.4 ms; parked=4 ...
conversation cache: restored 217373 tokens (checkpoint) in 620.5 ms; parked=4 ...
prompt 226654 tokens = 217373 reused + 9281 read in 5712 ms ... 8 checkpoints
```

So a copy of the same conversation was parked at 225,122 tokens 13 ms earlier, the next request extends it by 1,532 tokens — and the engine restores a checkpoint 7,749 tokens *behind* the parked state (and after it, the conversation is left with 8 checkpoints instead of 12). The client sends requests every 2-6 s in bursts, so these copies churn (dropped/parked/superseded) fast.

**Context that may matter:** the conversation is screenshot-heavy (10 base64 images, one every few turns; the client is doing visual QA). The docs say a checkpoint "is used only when the prompt starts with exactly its tokens **and pictures**" (DETAILS.md, conversation cache section), so if picture validation can reject the newest turn-boundary checkpoints, the fallback would land on the deepest picture-clean one — but the exact 32768 spacing of the fallbacks looks more like the *grid* checkpoints ("every 16K prompt tokens") surviving while the *turn-start* checkpoints do not.

**Question.** Why would the turn-start checkpoints be unavailable for this conversation (so the engine falls back to the 16K/32K grid), and why would the newest parked copy's state not be used at the next request? Is the intent that a superseded copy keeps only the grid checkpoints, or that a fast supersede/park churn drops the per-turn ones? If it helps, I can re-run a smaller reproduction (a synthetic conversation with an image, extended a few turns, raw bodies attached) or instrument further.

**Impact.** This is our largest remaining TTFT cost for that client: ~2.5 minutes of re-reading across 13 requests in one 3.5 h session, each a 4-17 s stall that also blocks the single serving turn.