#6327 [Engine] Compaction re-reads 331K-342K tokens five times (19.5-20.1 min each): the cache budget is the blocker (PR #4338)
open · @xvcezrqpbxtpzblrt · 0 comments · View on GitHub
BenchmarksSetup & installServer & APIMulti-GPUAMD / HIPNVIDIA / CUDAModels & quantsDocumentationWindowsLinux
Description
## Fastest fix: size the conversation cache budget (PR #4338)
**The 20-minute read is a cache-budget problem, and the fix is already measured.** The auto budget is 3,840 MiB but one snapshot at ~340K is 5,742-6,926 MiB, so the conversation is never parked and the compaction request — itself a prefix of the conversation — is read from token 0. With the budget sized to hold one snapshot of `--max-context` (PR #4338), a real `/compact` reused 282,077 of 282,474 tokens (99.86%) and prefilled in 3.8 s. No KV format change, no re-anchor, no rebuilt GDN state and no new kernel is involved.
**A second path is already in the tree, off by default:** in `serve/server.py`, recognize a compaction request by its **request shape** (summarization instruction + empty tool list) and, on a hit, re-render it with the same conversation's **previous non-compaction request's system prompt and tool block** — the exact body the existing opt-in already produces. That covers the client-side case where the request's top tokens change. The two paths target the same regression and neither is a prerequisite for the other; see "Proposed fix" below for the single recommendation.
1. **`docs/DETAILS.md:1548-1561` already documents the solution.** Compaction re-sends the conversation with `tools: []`; the tool block sits at the top of the template, so the prompt shares only its first few tokens and is re-read from the start. The same section records the opt-in that re-renders such a request with the previous conversation's tools: with it, **84,895 / 84,997 tokens reused (99.88%), ~100 re-read**; without it, **41 / 80,683 reused** — a full re-read. That is the same 20-minute-read problem, already measured upstream.
2. **`serve/server.py:3831, 7351-7373, 8491` is the only compaction handling in the tree** and it is that opt-in, **off by default**. A search of `src/program/generate.cpp`, `core/conversation_*` and `kernels/*.hpp` for `compaction` returns nothing: the engine has no compaction request type, so it cannot retain the previous tools or a parent handle. The server-side re-render is the only mechanism that exists, and most clients simply never enable it.
3. **Minimal patch (merge-ready):**
- **a.** In `serve/server.py`, detect compaction by request shape; on a hit, render with the previous non-compaction request's system prompt + tool block. Keep the existing response parsing (tools = absent). This generalizes the per-client opt-in to every client.
- **b.** Guard it: require that the previous request in the same conversation actually carried tools, and that the summarization instruction pattern matches, so a genuine "summarize with tools off" request is not affected.
- **c.** Client-side complement, zero engine code: move the summarization instruction to the **end** of the history instead of removing the tool block. The compaction request then becomes a strict extension of the previous prompt and reuses ~100%; any client adopting this removes its compaction-side cold read immediately.
4. **Neither path is gated on the KV work.** The segment-addressed KV cache (#3584) is the right long-term direction, but it is **not** a prerequisite for the compaction request's 20-minute read. Only the first turn after a summary needs it (see Acceptance criteria).
---
### Area
[Engine] (speed, kernels, models, the server and API)
### Desktop app, Terminal or source?
Desktop app (Windows, Linux or Mac).
### Which API path the desktop app actually uses for compaction (static analysis, not measured)
This matters for the request-shape re-render path described above: the desktop app does **not** compact through Codex's `/v1/responses/`, so the existing `codex_compaction_cache` opt-in (which only re-renders `/v1/responses/` compaction requests) never applies to it.
Reading the shipped app sources (unconfirmed static analysis - I did not instrument a live desktop run):
- Typing `/compact`, or an automatic compaction, ends in one RPC: `runtime.gw.rpc('commands/execute', { agentId: sessionId, line: '/compact', submittedAttachments: [] })` (`app/host/server/agent/agent-bridge.mjs:1505`, `:1561`; `/compact` recognised in `app/host/server/index.mjs:657,660`). Success is `result.kind === 'success'` with text starting `Compacted`.
- The compaction itself is produced by the Harness plugins `@deepseek-ai/dsh-compaction-basic` and `@deepseek-ai/dsh-command-compact` (`app/host/server/agent/preset.mjs:96-97`). Their source is not in this tree, so the exact summarization instruction wording could not be verified here.
- `app/host/server/agent/plugins/strata-context.mjs` wraps `llm.stream` and intercepts requests marked `purpose: 'compaction'`. When the request fits the window it is forwarded **as is** (`:168-170`), so the conversation cache can serve it; when it does not fit, the client splits it into pieces (`:189-192`). Each piece reaches the local engine at `/v1/messages` (`serve/server.py:7161-7164`). The server's own startup line lists `OpenAI: /v1/chat/completions, Anthropic: /v1/messages` and no `/v1/responses/`.
- So on the desktop app, prefix reuse on compaction hinges on the "forwards the whole request unchanged" branch in `strata-context.mjs:168` - exactly the case a parked snapshot larger than the cache budget breaks. The `codex_compaction_cache` opt-in in `serve/responses.py` is `/v1/responses/`-only and never sees this traffic.
Consequence for the fix: the budget fix (PR #4338) stays the single recommendation - it is dialect-independent and applies to this traffic. If the request-shape re-render is implemented, it must sit on the `/v1/messages` (and `/v1/chat/completions`) path, not only in `responses.py`, to help the desktop app this report was filed from. That is a static-analysis conclusion; I did not measure a live desktop compaction here.
### What happened
A 512K-context agent session on engine 0.1.48 compacted five times in this log. Each compaction cost two full prompt reads:
- the compaction request itself re-read 331,476 to 342,496 tokens in 1,169,295 to 1,207,770 ms (19.5 to 20.1 minutes);
- the first turn after the summary re-read 95,686 to 103,519 tokens in 326,574 to 355,919 ms (5.4 to 5.9 minutes).
The five cycles alone re-read 2,174,210 tokens and spent 7,597,336 ms (126.6 minutes). Turns that continue the same prompt are cheap by comparison: 417,938 of 418,637 reused in 2.4 s, 411,833 of 412,172 in 1.6 s, 419,067 of 419,250 in 1.2 s. The cost is in the prompt shape, not in the model or the card.
The same rule explains other cold reads in the log. Four turns that were not compactions reused only the 6,165-token system prompt (or 0), so the prompt had changed at the top; the engine re-read 405,494 to 411,506 tokens in 1,436,198 to 1,458,463 ms (23.9 to 24.3 minutes each). One of them is the 411,659-token turn in Logs. The log does not record what changed. [inference: the tool block at the top; the next turn reused the new prompt, 411,833 of 412,172]
In total, these fourteen turns re-read 3,811,680 tokens and spent 13,376,812 ms (223.0 minutes) on prompts the engine had already read.
### Smallest steps or command that reproduce it
1. Start the engine with `--max-context 524288 --kv int8` on a 16 GB card, model Swift 1.5 IQ3_XXS, 6 checkpoints per request.
2. Use an agent client that compacts by re-sending the conversation with a summarization instruction and an empty tool list; `docs/DETAILS.md:1548-1561` describes this request shape and says the tool block sits at the top of the template.
3. Let the session grow to about 419K tokens so the client compacts.
4. Read the `prompt ... tokens = ... reused + ... read in ... ms` line of the compaction request: 0 or 6,127 reused, 331K-342K read, 19.5-20.1 minutes.
5. Read the line of the first turn after the summary: 0 reused, 95K-104K read, 5.4-5.9 minutes.
### Reproduction status
Reproduced locally: five compactions in one session, read from the engine log. The log spans two engine runs of the same engine version. Not a controlled benchmark, and not repeated on a second machine. Additionally reproduced on current engine 0.1.59 at 384K context — see "Reproduced on current engine 0.1.59 at 384K context" below.
### Strata version
The five compactions were on engine 0.1.48, Windows x86-64 CUDA build. The same cold read was later reproduced on 0.1.59 (384K) and re-tested on 0.2.16 (see below); the two budget measurements cited in this report were on engine 0.1.50. Code and documentation line numbers are from the public tree as of 2026-10-24; the installed engine was built earlier, so the lines may shift, but the log strings match.
Re-tested on Strata 0.2.16 (2026-11-19), engine 0.1.62, Windows 11 Pro, same 384K configuration (`--max-context 393216`, Qwen3.8-Flash-Next GSQ-RCO IQ3_S, `--kv k8v4`). The from-first-unit re-read reported here is still present in the released source: `engine/include/strata/core/conversation_cache.hpp:248` still matches the cached conversation only from the first unit (`std::equal(c.ids.begin(), c.ids.end(), prompt.begin())`), and `terminal/serve/server.py` has no structural anchoring. On the real 0.2.16 engine, an unanchored compaction resend still cannot reuse the cached prefix; with a locally installed structural-anchor patch the compaction resend reuses 6560 of 6565 prompt units and re-reads only 34 (verbatim `strata serve` log in the 2026-11-19 update on #6232). The 0.2.16 release notes do not mention compaction prefix reuse, and this defect is not fixed there.
### Graphics card, memory and system
NVIDIA GeForce RTX 3080 Ti Laptop GPU, 16 GB VRAM, compute capability 8.6 · 64 GB RAM (63.8 GiB guard limit) · Windows 11 Pro · 16 physical cores, 24 threads. GPU model re-verified locally via `Get-CimInstance Win32_VideoController`: `NVIDIA GeForce RTX 3080 Ti Laptop GPU`.
### Model, size and settings
Swift 1.5 IQ3_XXS (GSQ-RCO IQ3_XXS) · `--max-context 524288` · `--kv int8` · `--rope-scaling yarn --rope-scale 2` · `--vram-reserve-mib 1536` · `--spec 4` · conversation cache auto-sized to 3,840 MiB for up to 4 parked conversations · 6 checkpoints reported per request.
Second configuration (0.1.59 reproduction): Qwen3.8-Flash-Next GSQ-RCO IQ3_S · `--max-context 393216` (384K) · `--kv k8v4` · 6 checkpoints per request.
### Logs
One full cycle (the fifth compaction):
```text
strata serve: prompt 419250 tokens = 419067 reused + 183 read in 1202 ms (152.3 tok/s), 696 generated in 15334 ms (45.4 tok/s), drafts accepted 530 of 578, 6 checkpoints
strata serve: conversation cache: skip parking (snapshot 6910 MiB exceeds available budget)
strata serve: prompt 339637 tokens = 0 reused + 339637 read in 1207770 ms (281.2 tok/s), 7017 generated in 226085 ms (31.0 tok/s), drafts accepted 4187 of 6706, 6 checkpoints
strata serve: conversation cache: skip parking (snapshot 5841 MiB exceeds available budget)
strata serve: prompt 103519 tokens = 0 reused + 103519 read in 355919 ms (290.8 tok/s), 1146 generated in 29149 ms (39.3 tok/s), drafts accepted 710 of 1049, 6 checkpoints
strata serve: prompt 105096 tokens = 104665 reused + 431 read in 1640 ms (262.8 tok/s), 958 generated in 25802 ms (37.1 tok/s), drafts accepted 574 of 878, 6 checkpoints
```
The other four compaction requests and the first turn after each:
```text
strata serve: prompt 337603 tokens = 6127 reused + 331476 read in 1169295 ms (283.5 tok/s), 2235 generated in 73130 ms (30.6 tok/s), drafts accepted 1299 of 2085, 6 checkpoints
strata serve: prompt 96868 tokens = 0 reused + 96868 read in 329723 ms (293.8 tok/s), 6834 generated in 168697 ms (40.5 tok/s), drafts accepted 4291 of 5622, 6 checkpoints
strata serve: prompt 336001 tokens = 0 reused + 336001 read in 1170580 ms (287.0 tok/s), 4168 generated in 119715 ms (34.8 tok/s), drafts accepted 2450 of 3921, 6 checkpoints
strata serve: prompt 99016 tokens = 0 reused + 99016 read in 336779 ms (294.0 tok/s), 301 generated in 7180 ms (41.9 tok/s), drafts accepted 206 of 262, 6 checkpoints
strata serve: promRelated on this site
Quick links to related pages.