#6627 [Engine] Inkling-2-Mini: multi-turn chats never reuse the prefix, so every turn re-reads the whole conversation (hidden thinking is in the slot but never in the next prompt)

open · @martinshane · 2 comentários · No GitHub

BenchmarksSetup & installServer & APIModels & quants

Descrição

### Area

[Engine]  (speed, kernels, models, the server and API)

### Desktop app, Terminal or source?

Terminal (the `strata` command)

### What happened

Every follow-up turn in a conversation re-reads the whole prompt from scratch. On a ~22.9K-token conversation each reply takes ~55 s on this Mac, even though the new prompt is the previous prompt plus the previous answer (replayed verbatim, `finish_reason: stop`) plus a short question. For agent clients, which resend a growing 10-40K-token history on every step, that is a full cold read per step.

Measured with the OpenAI endpoint, temperature 0, one slot (the default):

| turn | prompt_tokens | cached_tokens | wall |
|---|---:|---:|---:|
| 0 (cold, 22.9K) | 22,853 | 0 | 61.7 s |
| 1 (+ answer + short question) | 22,918 | 0 | 55.7 s |
| 2 (+ answer + short question) | 22,937 | 0 | 56.6 s |

`prompt_tokens_details.cached_tokens` is 0 on every turn, and `GET /metrics` showed `reused: null` on every request of a second 3-turn run (see Logs). Cold prefill is ~440-470 tok/s here, so ~55 s is a full re-read, not a partial one.

**What I think causes it (from reading the code, please correct me):**

1. The model writes hidden reasoning even at the forced effort `none`: a one-word answer ("Red") reports `completion_tokens: 51`. The server drops that span on purpose ([inkling_dialect.py#L189](https://github.com/Niko1221/Strata/blob/c6f6427d9de9016fd92d13e29740f65625261df7/terminal/serve/inkling_dialect.py#L189), and `template_kwargs` forces `reasoning_effort="none"`), so the client never receives it.
2. The engine's slot keeps prompt + **every generated token, the hidden reasoning included** in `cached_prefix`, and reuses only when that WHOLE list is a prefix of the next prompt; any partial match resets to 0 because SWA/ShortConv state can't rewind ([ink_serve.h#L475-L487](https://github.com/Niko1221/Strata/blob/c6f6427d9de9016fd92d13e29740f65625261df7/engine/src/inkling2/ink_serve.h#L475-L487)).
3. The next prompt contains the visible answer but not the hidden reasoning, so the match always fails at the start of the previous answer, and reuse drops to 0 instead of to the end of the previous prompt.

So the reuse can never fire in a normal multi-turn chat. An exact repeat of the same request doesn't reuse either.

**Expected:** a follow-up that extends the previous prompt reuses at least the previous prompt (the KV/state at the end of the previous prompt), as the Qwen path's conversation cache does. A possible fix: snapshot the slot's SWA/ShortConv state at the end of the prompt (before generation) and resume from that point when the next prompt matches up to there. Alternatively, return the reasoning so clients can replay it, but agent clients usually drop reasoning from history, so the snapshot is the robust fix.

Single-request numbers are great (prefill ~450 tok/s, the 4-task agent check I use passed as well as my current local default in half the time), which is why I'd like to use it as an agent model.

### Smallest steps or command that reproduce it

1. `strata install --family inkling-2-mini --model SQ-15 --context 65536 --vision yes --no-start --yes`
2. `strata launch --model sq-15 --family inkling-2-mini --context 64k --vision on --tune skip --access local --port 8190 --no-tui`
3. POST /v1/chat/completions with a ~20K-token user message ending in a question; wait for `finish_reason: stop`.
4. POST again with the same messages + `{"role":"assistant","content":<the exact answer>}` + a short new user question.
5. Step 4 has `prompt_tokens_details.cached_tokens: 0` and takes as long as step 3's prefill. Repeat for more turns: same every time.

Quick check that hidden reasoning is generated: ask "Name a color. One word." -> content "Red", `completion_tokens: 51`.

### Reproduction status

Reproduced locally

### Strata version

Strata 0.2.19 · Terminal build 0.1.65 · strata-inkling 0.1.65 (macos-arm64-metal)

### Graphics card, memory and system

Apple M4 Pro (MacBook Pro) · 48 GB unified memory · macOS 26.6.2

### Model, size and settings

Inkling-2-Mini SQ-15 + Vision pack · 64K context · pictures on · DSpark on (defaults)

### Logs

```
Engine log (strata launch):
[strata] engine started: .../components/shapes/inkling2/engine/0.1.65-macos-arm64-metal/strata-inkling --serve --native .../Inkling-2-Mini-SQ-15.strata --max-context 65536 --vision .../Inkling-2-Mini-Vision.strata --draft dspark
[ink_metal] loaded .../Inkling-2-Mini-SQ-15.strata (build=SQ-15, minor=1, 13.36 GiB, trunk=646, dspark=54, experts=3072, vram_order=3072) in 0.09s (workSet=37.4 GiB, maxBuf=28.1 GiB)
[ink_metal] runner ready: slots=1 ctx=65536 batch=512 dspark=1 vision=1 mpp=1(trunk=1,hot=0) | KV=2208.0 MiB SC=3.7 MiB TotalAlloc=18.86 GiB

/metrics requests (newest first) for a second 3-turn run on a ~3.6K-token prompt (turns stopped at max_tokens 200), plus an exact repeat of the last request: reused null on all, prompt_tokens 3599 -> 3612 -> 3622 -> 3622, live.slots[0].held_tokens stayed at 14 (the warm-up request) throughout.
```

No site

Links install, modelos, releases.