#6655 [Engine] Large-context compaction re-reads history from token 0 when the snapshot cannot be parked
open · @xvcezrqpbxtpzblrt · 0 コメント · GitHub で見る
BenchmarksSetup & installServer & APIAMD / HIPNVIDIA / CUDAModels & quantsDocumentationWindows
本文
### Area [Engine] (speed, kernels, models, the server and API) ### Desktop app, Terminal or source? Desktop app (Windows). ### What happened Compaction in a long, large-context agent session can take minutes instead of reusing the history that is already in the conversation cache. I measured real, repeated host-level compaction loops (`POST /api/chat/stream`, one session, tokens accumulated across turns, 80% threshold) on the current release, Qwen3.8-Flash-Next GSQ-RCO IQ3_S. The result depends entirely on whether the pre-compaction snapshot can be **parked**: | Config | Round | pre-compact ctx | compaction time | snapshot parked? | |---|---|---:|---:|---| | 384K, k8v4, 18 GiB cache, keep 8 GiB RAM | 1 | 273,409 | **36.4 s** | yes | | same | 2 | 211,848 | **20.7 s** | yes | | same | 3 | 211,813 | **23.0 s** | yes | | 425,984, k8v4, default cache | 1 | 301,407 | **221.3 s** | no — refused, re-read from token 0 | | 1M, int8, `--mmap-experts` | 1 | 800,819 | **918.2 s** | no — refused, re-read from token 0 | When the snapshot parks, three consecutive compactions stay in the 20-36 s range and the session keeps cycling. When the physical-RAM admission gate refuses it, the engine returns **before** saving the snapshot and re-reads ~200K-800K tokens from token 0 on the next turn — 221 s at 425K and 918 s at 1M in a single cycle. The slowness is not model decode; it is the from-token-0 re-prefill caused by a refused park. Note on the numbers: the 20-36 s above is only the `compacting` -> `compacted` phase measured from the streamed frames. The ~500 s spent filling the window with test text before each round is the test harness, not compaction, and is not part of this report. ### Smallest steps or command that reproduce it 1. Start the engine on a 16 GB card with a large context, e.g. `--max-context 425984 --kv k8v4` (or 1048576 / int8 with `--mmap-experts`). 2. In an agent session that compacts at 80% of the window, grow one conversation past the threshold (~301K tokens at 425984, 80% = 340787). 3. Watch the engine log on the compaction request. When RAM at park time is tight, it prints `conversation cache: skip parking (physical RAM admission; need N MiB plus 1024 MiB floor, M MiB available ...)`. 4. Read the following `prompt ... tokens = ... reused + ... read in ... ms` line: few reused, ~all tokens read, ~221 s (425K) or ~918 s (1M). 5. Shrink the context to the sweet spot (384K / k8v4 / 18 GiB cache / keep 8 GiB RAM) and repeat: the snapshot parks (`conversation cache: parked ...`) and each of three rounds compacts in 20-36 s. ### Reproduction status Reproduced locally (real multi-round loops, three consecutive rounds on the current release). Not a controlled benchmark and not repeated on a second machine. Some code/line references below are unconfirmed static analysis from the public tree, labelled as such. ### Strata version Strata 0.2.19 — engine 0.1.65-windows-x86_64-cuda-cuda13. Verbatim from the log: `strata generate: session is up (engine 0.1.65)`. ### Graphics card, memory and system NVIDIA GeForce RTX 3080 Ti Laptop GPU, 16 GB VRAM, compute capability 8.6 · 64 GB RAM (63.7 GiB guard limit) · Windows 11 Pro · 16 physical cores, 24 threads. Model re-verified with `Get-CimInstance Win32_VideoController`. ### Model, size and settings Qwen3.8-Flash-Next GSQ-RCO IQ3_S · sweet-spot config: `--max-context 393216` (384K), `--kv k8v4`, `--kv-resident 32768`, `--rope-scaling yarn --rope-scale 1.5`, `--conversation-cache-mib 18432`, `--conversation-cache-min-free-mib 1024`, `STRATA_RAM_KEEP_GIB=8`, `--spec 6`, 6 checkpoints per request. Failure configs: 425,984 k8v4 and 1,048,576 int8 with `--mmap-experts`. ### Logs Real multi-round loop, 384K k8v4 — the snapshot parks and is reused (verbatim `strata serve` lines): ```text strata serve: conversation cache: parked 273408 tokens in 301.5 ms; parked=2 ... strata serve: conversation cache: spilled 273408 tokens to the SSD tier (evicted from RAM, written in the background) in 2559.8 ms; on SSD 5, 12249 MiB strata serve: conversation cache: evicted 2 parked conversations to admit this snapshot (3683 MiB plus 1024 MiB floor) strata serve: conversation cache: parked 267473 tokens in 5219.6 ms; parked=2 ... strata serve: conversation cache: parked 211847 tokens in 3805.4 ms; parked=3 ... strata serve: conversation cache: parked 211812 tokens in 4199.1 ms; parked=3 ... ``` Failure at 425,984 k8v4 — park refused, full re-read (verbatim): ```text strata serve: prompt 301407 tokens = 6493 reused + 294914 read in 312075 ms strata serve: conversation cache: skip parking (physical RAM admission; need 4353 MiB plus 1024 MiB floor, 3940 MiB available ...) ``` Failure at 1M int8 mmap — park refused, full re-read (verbatim): ```text strata serve: conversation cache: skip parking (physical RAM admission; need 12463 MiB plus 1024 MiB floor, 3166 MiB available; evicted 0, 2 still parked ...) strata serve: prompt 800819 tokens = ... read in 918180 ms ``` Log file: `strata-iq3_s.log` from the Strata install folder. (API keys/passwords/private text removed; none appeared in these lines.) ### Why this is worth fixing for everyone This is a general mechanism, not one machine or one quant. The part that decides the outcome is whether `available RAM - 1 GiB floor` can hold the snapshot at the moment of parking. The larger the context and the longer the prefill in mapped/low-RAM mode, the less RAM is free right when the park is attempted — exactly when a long session compacts. A bigger cache budget alone does not help the physical-RAM gate. So any user who grows a session near their RAM ceiling hits the same from-token-0 re-read; machines with lots of headroom simply never see it. This is why the same session compacts fine early and starts taking minutes once it gets long. ### Root cause (two independent gates) 1. **Cache budget gate** — the auto budget is 40% of RAM above the floor, capped at 8 GiB. One snapshot at ~300K k8v4 is ~4-6 GiB and at 1M int8 is ~12.4 GiB, so it can exceed the budget and never park. Explicit `--conversation-cache-mib 18432` passes this gate. 2. **Physical-RAM admission gate** — even with a large budget, a long prefill in mapped/low-RAM mode leaves little free RAM at park time, so `conversation_memory_admit` refuses the snapshot. The engine returns **before** `conversation_snapshot_save` (static analysis: `generate.cpp:13904-13910` vs the save at `:13938`), so nothing is captured and the next turn re-reads from token 0. The expert file pages read during prefill are charged to `available` in `sample_host_memory()` (static analysis: `conversation_memory.cpp:411-421`), so a long prefill drives the reading down. The project already ships the fix for the second gate but leaves it opt-in and off by default: `STRATA_FILE_RELEASE=1` trims those file pages after GPU upload. The project's own Windows benchmark (`docs/bench-results/2026-10-04-windows-mapped-release/README.md:81`) measures median available RAM rising from 0.51 to 12.19 GiB at a 3.7-4.4% prefill cost. ### Proposed fix Two changes, both general and none tied to this configuration: 1. **Make the file-page release automatic on park failure.** When the physical-RAM admission gate refuses a snapshot in mapped mode, run the `STRATA_FILE_RELEASE` trim as part of the admission path before giving up, instead of requiring the user to set the flag in advance. 2. **Fall back to the existing SSD tier instead of dropping the history.** On gate failure, capture the snapshot into the SSD spill tier and restore the prefix on the next request, rather than returning before `conversation_snapshot_save` and discarding everything already read. The SSD tier already exists and is used in the successful 384K loop above. 3. **Count locked weights and reclaimable file pages separately** in the admission decision, so reclaimable cache does not make the engine think RAM is exhausted. Recommended acceptance check: with a large context (e.g. 425,984 k8v4, or 1M int8 mmap) on a 64 GB machine, the snapshot parks with no `skip parking (physical RAM admission)` line and several consecutive compactions each reuse >=95% of the pre-compaction history without a from-token-0 re-read. ### Duplicate search Searched open and closed issues for `skip parking`, `compaction context`, `conversation cache`, `parked`, `re-read`, `budget`. Closest: - **#5895 (open)** — same 64 GB machine, a 260k conversation cannot be parked, every side request forces a full re-read (follow-up to #57 / #190). Closest match; this report adds real multi-round compaction-loop numbers and the two-gate/FILE_RELEASE mechanism. - **#5515 (open)** — in-place retune (TUNE) and per-shape conversation-cache routing. Related to budget/admission sizing. - **#2255 (closed)** — refused parking has no counter and no reason split. - **#6327 (closed)** — an earlier compaction re-read report on this machine (single-shot, superseded by the real multi-round numbers above). - **#3928 (closed)** — the Stop-path variant: a cancelled read loses its prefix and the next request re-reads 375K tokens. I found no open report that measures several consecutive compaction loops on the current release and isolates the physical-RAM admission gate as the deciding factor.
関連リンク
インストール・モデル・リリースへのサイト内リンク。