#6508 [Engine] q36: resume from the longest common prefix with GDN state checkpoints — 0 → 12,288 reused, 37.1 → 2.9 s
open · @Mark-1982 · 0 コメント · GitHub で見る
Server & APIAMD / HIPNVIDIA / CUDA
本文
## Problem and result
On the `qwen35moe` / `strata-q36` path, reuse between requests was all-or-nothing: `match_len` required the session's **entire** held run — the prompt *and* the answer it generated — to be a proper start of the arriving prompt, because the GDN layers keep only the recurrent state as it stands at the end of that run and cannot step back. One changed byte early in a conversation therefore cost the whole prompt, not the tail after it.
That is the normal case for an agent client, not an edge case: any client that re-renders its transcript each turn can rewrite an *earlier* assistant turn while the tail still matches. Hermes normalises tool-call argument JSON on the send path, so `{"path":…,"limit":…}` comes back as `{"limit":…,"path":…}` and, with Qwen3.6's template, two `<parameter=…>` blocks swap places 11,624 characters into an already-generated turn.
Same body, same session, same binary, only `STRATA_Q36_CKPT_N` changing:
| | prompt | reused | read | time |
|---|---:|---:|---:|---:|
| before | 12,680 | **0** | 12,680 | 37,125 ms |
| after | 12,680 | **12,288** | 392 | **2,851 ms** |
A live agent session (Hermes, 3 tool calls) on the same rung went from `0 reused + 38026 read in 108414 ms` to reusing on every turn: `12049 = 8192 reused + 3857 read`, …, `12487 = 8192 reused + 4295 read`. Reuse lands on a checkpoint multiple rather than exactly at the divergence, which is the intended tradeoff: snapshot interval against host memory.
## Related discussion
Closes #6507.
- **#6507** — this problem, filed first with the measurements above.
- **#5145** (open) — the same engine limitation from the other side: `strata_prefix` / `pin=N` is parsed away on q36, so a *shared* prefix across independent requests is re-read. Its "Ask" item 1 asks for exactly this mechanism — "snapshot the session's state at token N … and on a later request whose ids start with the same N tokens, `copy_session` from it and read only the rest". This PR builds the snapshot-and-resume half; it does **not** add `pin=N` parsing or a cross-session prefix buffer, so #5145 is not closed by it. `pdmaclennan-art`'s comment there ("A state snapshot at the end of the prompt … would cover the agent case too") is the case measured above.
- **#2840** (closed, not planned) — the original conversation-cache request for this path.
- Re-read-cost family, different mechanism (budget and eviction rather than the reuse rule), not addressed here: #6327, #6232, #6469, #5895, #6463.
No scope agreement was requested in advance; #6507 and this PR were opened together so the mechanism is visible next to the problem. Close either if you would rather shape it differently — the measurements stand on their own.
## Approach
`Engine` gains a ring of `ckpt_n` host snapshots per session. Each holds every GDN layer's `conv` and `S`, plus `hlast`, in pinned memory (`cudaHostAlloc`), sized from the runtime's own geometry — nothing hardcoded:
```
conv = ssm_conv_channels * (ssm_d_conv - 1) * 4
S = ssm_v_heads * ssm_state_size^2 * 4
hlast = n_embd * 4
```
On this model that is 30 GDN layers × (98,304 + 2,097,152) B + 8,192 B = **62.82 MiB** per checkpoint, 251.28 MiB for a ring of 4 per session. The attention layers need nothing saved: their KV cells before the resume point are already correct.
- Snapshots are taken when the session crosses a multiple of `ckpt_every`, both while reading a prompt and while producing tokens, once per multiple, with a stream sync so the copy is of settled state.
- `match_len` now returns the **longest common prefix** length instead of 0-or-everything. For a vision session the prefix is pulled back to a picture boundary, so a picture is never half-reused.
- The resume point is the newest checkpoint at or below that length (clamped to leave at least one token to read, so the last-token path is unchanged). Below the lowest checkpoint the engine resets exactly as before.
- `copy_session` carries the ring; `reset` drops it. `pick_session` prefers a session that can actually resume.
- The serve log line gains `, N ckpt` so the behaviour is visible in logs.
`hlast` travels with `conv`/`S` so the MTP block's seed is right after a resume; without it the first drafted token after a resume differs.
## Validation
**Passed**
- `STRATA_Q36_CKPT_N=0` is byte-identical to the pre-change binary. 512 tokens, temperature 0, top_p 1, identical request bodies (`sha256 1112582a73ef11e7…`), two runs each build: all four outputs `sha256 c4f343b7cfee9215…`, and no `ckpt` line appears in the log. Measured on a single-card rung, because the two-card rung on this box is nondeterministic run-to-run independently of this change (the pre-change binary alone gave three different texts in three runs: 1,911 / 1,906 / 1,892 chars).
- Resume fidelity, the check that matters most: on the same 12,680-token body, a **checkpoint resume at 12,288 reproduces the cold-read generation byte for byte** (both `len=62 sha=112060777dbe36131b0cd497d80408f8`). The restored state is the prefix's real state, not merely a non-zero counter.
- The table above (`0 → 12,288 reused`), and an identical prompt sent twice: `12,288 reused + 421 read`, was `0 reused`.
- 10-turn agent soak, prompts 43,903 → 70,417 tokens: no crash, no `cudaMalloc` failure, no error lines; reuse climbed 28.0% → 95.9%; 11,529 / 11,309 MiB on the two cards, 70.76 GiB `memory.current`.
- Compiles clean against `main` at 3142b66: `nvcc -O3 -std=c++20 --generate-code=arch=compute_86,code=[compute_86,sm_86]`, CUDA 13.1, rc 0, 1,798,168 B object. Only pre-existing warnings (unused `v`, `D`, `k_add`, `k_gate_rows`).
**Failed**
- None observed.
**Not run**
- **HIP, SYCL and Metal are not built.** This file is CUDA-only in this tree and the change touches no shared header, but I have no AMD, Intel or Mac hardware, so CONTRIBUTING's "build every backend a file touches" is met only for CUDA. Please treat that as unverified.
- The two-card `--peer-device` rung could not serve as the flag-off identity control (nondeterminism above), so gate 1 is single-card only.
- No `--batch` / multi-row decode coverage: `run_rows` does not checkpoint (see Limits).
- The engine measurements were taken on a build that also carries our `--peer-device` (#6425) and `--kv fp8` (#6430) patches, because this box cannot hold Q8_0 on one card. The before/after pair differs only in `STRATA_Q36_CKPT_N`, and the upstream-clean build above compiles and is what this PR contains.
## Limits
- Reuse resumes at a checkpoint multiple, not at the exact divergence, so up to `ckpt_every - 1` tokens are re-read even though the state for them existed. Smaller intervals cost proportionally more pinned host memory.
- 251.28 MiB pinned per session at the defaults (×4 at 1024-token intervals). Pinned memory is not swappable; on a small-RAM machine `STRATA_Q36_CKPT_N=0` or a shallower ring is the answer.
- A `--batch` decode window (`run_rows`) does not take checkpoints — single-session paths only.
- No `pin=N` / `strata_prefix` support and no cross-session prefix buffer, so #5145's own case is untouched.
- Defaults change behaviour for q36 users (reuse where there was none, plus the pinned memory). If you would rather ship it off by default, `STRATA_Q36_CKPT_N=0` is already the inert path — say so and I will flip the default and re-measure.
## AI assistance
Written with agent assistance and reviewed line by line against the source and the raw logs before submitting; every number above comes from a command run on the hardware described, not from a model's summary. I can explain any changed line and will answer review questions directly.
関連リンク
インストール・モデル・リリースへのサイト内リンク。