#6232 [Engine] 384K compaction re-reads 264K tokens (456 s): content-addressed, position-independent segment cache for near-zero-reread compaction
closed · @xvcezrqpbxtpzblrt · 1 comments · View on GitHub
BenchmarksSetup & installServer & APIMulti-GPUAMD / HIPNVIDIA / CUDAModels & quantsDocumentationWindowsLinux
Description
## Evidence status of this report
This report deliberately separates three kinds of statements, so nothing here is overclaimed.
- **(a) Verified facts** - verbatim excerpts and figures from primary sources (papers and official docs), plus measurements recorded in Strata's own docs or read directly from the engine log on this machine.
- **(b) Proposals** - the design in Parts 0-5 and Stages B-D below is **not implemented in Strata**. It is engineering inference built on (a).
- **(c) Targets** - near-zero-reread, >= 95% reuse, max abs diff <= 1e-3 and >= 95% crash recovery are **acceptance criteria / goals**, not achieved results. Where a third-party paper reports a speedup, that is the paper's result under its workload, not a promise for Strata.
The reporter has also built and locally installed a Stage A patch (see Suggested rollout). Its prompt-prefix measurement is a verified fact (category a) about what was measured on this machine; the patch itself is a candidate fix (category b), installed locally only and **not merged into Strata**. The official 99.88% figure quoted there belongs to Strata's existing `codex_compaction_cache`, not to this patch.
This continues #3998. That report was auto-closed in a stale-issue sweep ("closed does not mean rejected"), and its `/reopen` is currently blocked by the bot gate that demands a re-test on 0.2.0-beta. The local desktop app ships engine **0.1.59**, so I cannot satisfy that gate on this machine. I am therefore opening one complete proposal instead of fighting the gate: the same measured defect, on the current engine at 384K, **plus a concrete engine-level design that removes it for every client**. The title carries the required `[Engine]` area and the form below is filled per CONTRIBUTING.md.
**Why this matters beyond a benchmark:** compaction latency is what makes large contexts unusable in practice. This machine can run 384K, but one compaction re-reads ~264K tokens from token 0 and stalls for minutes, so the user is pushed to pick a smaller context just to avoid the pause. That is a mechanism defect, not a hardware limit - it throws away capacity the hardware already has.
## Area
[Engine] (speed, kernels, models, the server and API)
## Desktop app, Terminal or source?
Desktop app (Windows, Linux or Mac).
## What happened
On the current engine at a **384K context**, a compaction request re-reads the entire history from token 0. Measured, verbatim (engine 0.1.59, `--max-context 393216`, `--kv k8v4`):
- `prompt 270500 tokens = 6531 reused + 263969 read in 456380 ms` - 7.6 min to read 263,969 tokens; the only reuse is the 6,531-token system prompt.
- `prompt 280145 tokens = 6531 reused + 273614 read in 443010 ms` - same pattern, 7.4 min.
- `prompt 253327 tokens = 0 reused + 253327 read in 392028 ms` - 0 reuse.
- `conversation cache off: not enough free RAM to park conversations` and `conversation cache: skip parking (snapshot ... exceeds available budget)` - the cache appeared **529 times** in the same log as `skip parking`, so nothing survives to be reused.
This is the same failure mode as the original 512K report in #3998, where each compaction re-read 331K-342K tokens in 19.5-20.1 min and the first turn after the summary re-read 95K-104K tokens in 5.4-5.9 min - **126.6 min across five cycles**. Normal turns that keep the same prompt are cheap by comparison (417,938 / 418,637 reused in 2.4 s). The cost is in the prompt shape, not the model or the card.
**Root cause (measured from the current tree):**
1. The cache matches **only from token 0**. `conversation_prefix()` is at `include/strata/core/conversation_cache.hpp:243`; the comparison `std::equal(c.ids.begin(), c.ids.end(), prompt.begin())` is at `:247`. It returns 0 unless the checkpoint ids are an exact prefix of the new prompt. A change anywhere in the first N tokens invalidates every checkpoint of length >= N.
2. Compaction breaks the prefix **twice**: the compaction request re-sends with `tools: []` and the tool block sits at the top (so reuse stops after the system prompt); the turn after the summary starts with `system + summary + retained tail`, where the tail is byte-identical but shifted to new positions, so prefix-only matching reuses 0.
3. The engine has **no compaction request type**: it cannot tell a compaction from an ordinary prompt, so it cannot keep the previous tools or a parent handle. The only handling is the per-client, default-off `codex_compaction_cache` re-render: `docs/DETAILS.md:509` (listed off), `:924-931` (mechanism), `serve/server.py:4914` (initialized False), `:8979-8997` (rendering, carrying the `#924` comment), `:10442` (read from config as an opt-in).
4. The parked snapshot is **refused** (`skip parking`, 5,742-6,926 MiB snapshot vs 3,840 MiB budget), so the pre-compaction entry - and the root checkpoint the first post-summary turn could have borrowed - is gone.
5. The KV is **one positional image** (`ConversationKv`: cells + contiguous buffer, position implicit in cell index). A retained tail cannot be mounted at a new offset without re-applying position encoding and rebuilding the GDN recurrent state (`include/strata/kernels/gdn.hpp`, 48 layers) and indexer rows at the boundary.
## Smallest steps or command that reproduce it
1. Start the engine with `--max-context 393216 --kv k8v4` on a 16 GB card (here RTX 3080 Ti Laptop, 64 GB RAM), model Qwen3.8-Flash-Next GSQ-RCO IQ3_S.
2. Use an agent client that compacts by re-sending the conversation with a summarization instruction and an empty tool list (`docs/DETAILS.md:924-931` describes this shape; the tool block is written at the top of the template).
3. Let the session grow to ~264K-280K tokens so the client compacts.
4. Read the `prompt ... tokens = ... reused + ... read in ... ms` line of the compaction: ~6,531 reused, ~264K read, 443-456 s.
5. Read the line of the first turn after the summary: 0 reused, full tail re-read.
## Reproduction status
Reproduced locally: compaction and re-read read directly from the engine log on the current engine 0.1.59 at 384K. Not a controlled benchmark and not repeated on a second machine; the 512K multi-cycle measurements are recorded in #3998.
## Strata version
Strata desktop (latest Windows build) 璺?engine `0.1.59-windows-x86_64-cuda-cuda13`. (GitHub currently shows Strata 0.2.16; this desktop install ships engine 0.1.59 - hence the /reopen gate could not be met.)
## Graphics card, memory and system
NVIDIA GeForce RTX 3080 Ti Laptop GPU, 16 GB VRAM 璺?64 GB RAM (63.7 GiB guard limit) 璺?Windows 11 Pro 璺?12th Gen Intel i7-12800HX, 16 physical cores / 24 threads. GPU model re-verified locally via `Get-CimInstance Win32_VideoController`.
## Model, size and settings
Qwen3.8-Flash-Next GSQ-RCO IQ3_S 璺?`--max-context 393216` (384K) 璺?`--kv k8v4` 璺?6 checkpoints per request 璺?conversation cache off (not enough free RAM to park). Earlier 512K run in #3998: Swift 1.5 IQ3_XXS, `--max-context 524288`, `--kv int8`.
## Logs
Only the performance lines are included. Raw logs are not attached because they contain private conversation text; these lines are pure token/timing data. API keys and credentials are not included anywhere in this report.
```text
strata serve: prompt 270500 tokens = 6531 reused + 263969 read in 456380 ms (578.4 tok/s), 1517 generated in 44264 ms (34.3 tok/s), drafts accepted 12 of 33, 6 checkpoints
strata serve: prompt 280145 tokens = 6531 reused + 273614 read in 443010 ms (617.6 tok/s), 506 generated in 14854 ms (34.1 tok/s), drafts accepted 0 of 0, 6 checkpoints
strata serve: prompt 253327 tokens = 0 reused + 253327 read in 392028 ms (646.2 tok/s), 5219 generated in 178550 ms (29.2 tok/s), drafts accepted 658 of 1059, 6 checkpoints
strata serve: conversation cache off: not enough free RAM to park conversations
strata serve: conversation cache: skip parking (snapshot ... exceeds available budget)
```
The existing opt-in's result is recorded by Strata in `docs/DETAILS.md:1563`: with it, a compaction of an ~85K conversation reused 84,895 / 84,997 tokens (99.88%); without it, 41 / 80,683. Scope of that figure (category a): it is Strata's official measurement on **one client, one quantization, one RTX 5070** - not a universal guarantee. Log file: `strata-<model>.log`.
---
## Prior art verified against primary sources
All excerpts below are verbatim from primary sources checked for this report (category a). They are cited to ground the design, not to claim Strata already does this.
**1. StreamingLLM** - arXiv:2309.17453, https://arxiv.org/abs/2309.17453 , HTML https://arxiv.org/html/2309.17453v4 (MIT/Meta/CMU/NVIDIA, 2023; revised 2024-04).
- "keeping the KV of initial tokens will largely recover the performance of window attention"; "just 4 initial tokens sufficing"; "up to 4 million tokens"; "outperforms the sliding window recomputation baseline by up to 22.2鑴?speedup".
- "attention sink ... due to the strong attention scores towards initial tokens as a 'sink' even if they are not semantically important".
- Direct primary-source basis for Part 2's position re-encoding: "For encoding like RoPE, we cache the Keys of tokens prior to introducing the rotary transformation. Then, we apply position transformation to the keys in the rolling cache at each decoding phase." and "focuses on positions within the cache rather than those in the original text".
- Adopted in NVIDIA TensorRT-LLM, HuggingFace Transformers (SinkCache), Intel Extension for Transformers and MLC LLM.
- Relation: the pre-rotary-key cache plus position transform at use time is exactly the mechanism Part 2 proposes to adapt for a summary-shifted prompt. Difference: StreamingLLM shifts a rolling decode cache to extend context; a compaction must remount a retained tail behind newly generated summary text and rebuild GDN boundary state - not covered by the paper.
**2. SnapKV** - arXiv:2404.14469, https://arxiv.org/abs/2404.14469 , HTML https://arxiv.org/html/2404.14469v2 (UIUC/Cohere, 2024).
- "SnapKV automatically compresses KV caches by selecting clustered important KV positions for each attention head"; "3.6x increase in generation speed and an 8.2x enhancement in memory efficiency compared to the baseline when processing inputs of 16K tokens"; "process up to 380K context tokens on a single A100-80GB GPU"; "fine-tuning-free".
- Method: an observation window at the end of the prompt, voting C = sum of W_obs, Top-k selection, 1D pooling clustering, then the selected prefix KV concatenated with the observation window.
- Relation: evidence that keeping a selected subset of KV positions preserves quality. Not directly adopted: SnapKV changes which tokens are attended (a quality/length trade-off), whereas a Strata compaction is meant to preserve the exact conversation; it is complementary, a different objective.
**3. CacheBlend** - arXiv:2405.16444, https://arxiv.org/abs/2405.16444 , HTML https://arxiv.org/html/2405.16444v3 (UChicago/Microsoft, EuroSys'25 Best Paper, DOI 10.1145/3689031.3696098; also accepted at ACM TOCS). Code: github.com/LMCache/LMCache.
- "reuses the pre-computed KV caches, regardless prefix or not, and selectively recomputes the KV values of a small subset of tokens to partially update each reused KV cache"; "reduces time-to-first-token (TTFT) by 2.2-3.3x and increases the inference throughput by 2.8-5x from full KV recompute without compromising generation quality".
- Stated limitation: "cross-attention ... cannot be pre-computed as the preceding chunks are not known in advance".
- Relation: direct basis for Part 1 (mount chunks regardless of prefix) and Part 4 (recompute only a small subset - the appended instruction). Its stated limitation maps onto this case: in a compaction the new summary precedes the retained tail, so cross-attention into the tail depends on new text and that boundary is exactly what must be recomputed/rebuilt.
**4. vLLM Automatic Prefix Caching** - official doc https://docs.vllm.ai/en/latest/design/prefix_caching/Related on this site
Quick links to related pages.