#6652 Hy4-Flash: prompt/KV cache always resets to 0 in multi-turn agentic loops (--session-reuse 1 missing + strict memcmp on BPE re-tokenization)

open · @mastermk · 1 Kommentare · Auf GitHub

needs-format

Beschreibung

### Area

[Engine]  (speed, kernels, models, the server and API)

### Desktop app, Terminal or source?

Source (START-HERE.bat, setup.sh or a build)

### What happened

Prompt/KV cache reuse is `0` (`RESUME 0`) across consecutive turns in Cline (~60k context multi-turn tool calls), forcing a full ~60k token re-prefill (~100 s) on almost every request.

Two separate issues cause this:

1. **`--session-reuse 1` is not passed by `serve/hy4_kind.py`**
   `strata-hy4.exe` initializes `session_reuse` to `0` by default unless `--session-reuse 1` is passed on the CLI (and rejects `--conversation-cache-mib` as `unknown arg`). `hy4_kind.engine_args()` does not pass `--session-reuse 1`, so session reuse stays disabled out of the box.

2. **`strata-hy4.exe` uses all-or-nothing `memcmp` instead of longest-common-prefix**
   Even with `--session-reuse 1` enabled, `strata-hy4.exe` checks whether the entire `saved_tokens` array (`prev_prompt + prev_generated`) matches `new_tokens[:saved_n]` via a single `memcmp`:
   - If **any** token differs, it discards the entire cache (`RESUME 0`) and re-prefills from token `0`.
   - In multi-turn tool-calling chats, re-tokenizing the assistant's previous message text with `tok.encode()` frequently merges BPE pairs differently than token-by-token generation (e.g., generating `108` tokens vs re-encoding `106` tokens).
   - Whenever those 1–3 trailing tokens in the assistant reply differ, the entire 60,000+ identical prefix before it is thrown away. In our `/api/requests` traces, only a turn where BPE happened to produce the exact same token count hit the cache (`60,730` reused in `0.989 s`); all others missed with `0` reused in `~100 s`.

**Suggested fix:**
- Pass `"--session-reuse", "1"` by default in `serve/hy4_kind.py` (`engine_args()`).
- In `strata-hy4.exe`, find the longest common prefix between `saved_tokens` and `new_tokens` instead of requiring a 100% `memcmp` match on `saved_n`.
- (Complementary in `serve/server.py`): for `hy4`, preserve the exact generated `raw_ids` from the previous turn and splice them into the next turn's prompt token IDs when continuing the same conversation prefix.

### Reproduction status

Reproduced locally

### Strata version

Strata 0.2.19 · strata-hy4 0.1.64 (engine 0.1.65)

### Graphics card, memory and system

NVIDIA GeForce RTX 5090 32 GB · 128 GB RAM · Windows 11

### Model, size and settings

Hy4-Flash-SQ-128 · 256K context · --draft dfly2

Mehr auf der Site

Links zu Install, Modellen, Releases.