#6507 [Engine] q36: one early byte change costs the whole prompt — all-or-nothing match_len has no GDN state to step back to (0 reused + 38026 read, 108 s)

open · @Mark-1982 · 1 Kommentare · Auf GitHub

BenchmarksSetup & installServer & APIMulti-GPUNVIDIA / CUDAModels & quantsWindows

Beschreibung

### Area

[Engine]  (speed, kernels, models, the server and API)

### Desktop app, Terminal or source?

Source (START-HERE.bat, setup.sh or a build)

### What happened

On the `qwen35moe` / `strata-q36` path, reuse between requests is all-or-nothing: `match_len` (`q36_engine.cu`, the reuse site at ~L3404 on `main`) requires the session's **entire** held run — prompt *and* the answer it generated — to be a proper start of the arriving prompt, because the GDN layers keep only the recurrent state as it stands at the end of that run and cannot step back. One differing byte anywhere in the conversation therefore costs the whole prompt, not the tail after it.

For an agent client this is the normal case, not an edge case. Any client that re-renders its transcript each turn can change an *earlier* part of the conversation while the tail still matches — Hermes normalises tool-call argument JSON on the send path (`sort_keys=True`), so an assistant turn the model itself produced as `{"path":…,"limit":…}` is echoed back as `{"limit":…,"path":…}`. With Qwen3.6's template, that renders as two `<parameter=…>` blocks in swapped order inside the already-generated assistant turn:

```
char divergence index 11624 of 11742 (the arriving prompt is otherwise identical)
emitted:  <parameter=path>…</parameter><parameter=limit>…</parameter>
sent:     <parameter=limit>…</parameter><parameter=path>…</parameter>
engine:   strata serve: session 0: prompt 11902 tokens = 0 reused + 11902 read
```

Observed on our own agent sessions before the change below:

```
request 3:  prompt 38026 tokens = 0 reused + 38026 read in 108414 ms
request 4:  prompt 39086 tokens = 38366 reused + 720 read in 3022 ms
```

Request 3 is the only turn in that session whose preceding assistant message carried a tool call with two or more arguments in non-alphabetical order; every other turn reused normally. A second trigger from the same family: a boolean argument renders `True` rather than `true`, which diverges at index 11845 of the same body.

The cost is not the rewrite — it is that a rewrite 11,624 characters in throws away the 12,288 tokens before it as well.

### Smallest steps or command that reproduce it

1. Start the q36 path with one session and watch the engine log.
2. Send a conversation of N tokens, let it answer.
3. Send the same conversation again with **one byte changed early in it** and the rest identical (swapping two tool-call argument blocks in an assistant turn is the realistic version; any early edit does it).
4. The log reports `0 reused + N read`, although everything after the edit is unchanged and the prefix before it is intact.

Control, same body, same binary, same session, with the change in the PR below:

```
before:  prompt 12680 tokens = 0 reused + 12680 read in 37125 ms
after:   prompt 12680 tokens = 12288 reused + 392 read in 2851 ms, 4 ckpt
```

### Reproduction status

Reproduced locally

### Strata version

Strata 0.2.15 · Terminal 0.1.61 · engine built from source, `main` at 12d491d

### Graphics card, memory and system

2x RTX 3060 12 GB (sm_86) · Xeon W-2133 · 72 GB RAM · Debian 13 in a Proxmox LXC, CUDA 13

### Model, size and settings

Qwen3.6-35B-A3B Q8_0 (37.8 GB, pinned Unsloth feed) · `--max-context 204800 --prefill 1024 --mtp 2 --cpu-threads 6 --expert-vram 0.1 --kv fp8 --peer-device 1` · one session

Disclosure: that command line includes two flags from our own open PRs (`--peer-device` from #6425, `--kv fp8` from #6430), because this box cannot hold the model on one card. Reuse accounting does not depend on either; the control pair above was measured on the same build with only the checkpoint knob changing.

### Logs

```shell
strata-q36: GDN checkpoints: 4 per session x 62.82 MiB pinned host (251.28 MiB in all for 1) every 4096 tokens; 30 GDN layers, conv 98304 B + S 2097152 B + hlast 8192 B each

strata serve: session 0: prompt 12680 tokens = 0 reused + 12680 read in 37125 ms (341.3 tok/s), 11 generated in 353 ms  [pre-change control]
strata serve: session 0: prompt 12680 tokens = 12288 reused + 392 read in 2851 ms (137.5 tok/s), 11 generated in 347 ms, 4 ckpt  [with checkpoints]

strata serve: session 0: prompt 12049 tokens = 8192 reused + 3857 read in 11403 ms (338.3 tok/s), 3 ckpt   [live agent, 3-tool-call turn]
strata serve: session 0: prompt 12487 tokens = 8192 reused + 4295 read in 14254 ms (301.3 tok/s), 4 ckpt
```

### Related, and why this is a separate report

- **#5145** (open) — same engine limitation reached from the other side: `strata_prefix` / `pin=N` is parsed away on q36, so a *shared* prefix across independent requests is re-read. Its "Ask" item 1 asks for a snapshot at token N; `pdmaclennan-art`'s comment there reports the agent symptom (18k prompts, 6 requests, all `0 reused`, 33–68 s). The ask here is narrower and needs no API surface: the session's own continuation after an early edit. Filed separately so the two can be closed independently; happy to have it merged into #5145 if you would rather keep one thread.
- **#2840** (closed, not planned) — the original request for a conversation cache on this path, from the 0.2.0-beta tracker reset.
- Re-read-cost family, different mechanism (budget/eviction rather than the reuse rule): #6327, #6232, #6469, #5895, #6463.

### Ask

Keep enough state to resume from the longest common prefix instead of discarding the session. Concretely: a small ring of host-side snapshots of every GDN layer's conv and `S` at fixed token multiples, and a `match_len` that returns the common prefix length with the resume point pulled back to the newest snapshot at or below it.

A patch that does this is in the PR linked below — opt-in, inert when off, 226 lines in one file. Filing the issue first per CONTRIBUTING; the PR is there to be reviewed or ignored as you prefer.

Mehr auf der Site

Links zu Install, Modellen, Releases.