#6652 Hy4-Flash: prompt/KV cache always resets to 0 in multi-turn agentic loops (--session-reuse 1 missing + strict memcmp on BPE re-tokenization)
open · @mastermk · 1 comentarios · En GitHub
Descripción
### Area [Engine] (speed, kernels, models, the server and API) ### Desktop app, Terminal or source? Source (START-HERE.bat, setup.sh or a build) ### What happened Prompt/KV cache reuse is `0` (`RESUME 0`) across consecutive turns in Cline (~60k context multi-turn tool calls), forcing a full ~60k token re-prefill (~100 s) on almost every request. Two separate issues cause this: 1. **`--session-reuse 1` is not passed by `serve/hy4_kind.py`** `strata-hy4.exe` initializes `session_reuse` to `0` by default unless `--session-reuse 1` is passed on the CLI (and rejects `--conversation-cache-mib` as `unknown arg`). `hy4_kind.engine_args()` does not pass `--session-reuse 1`, so session reuse stays disabled out of the box. 2. **`strata-hy4.exe` uses all-or-nothing `memcmp` instead of longest-common-prefix** Even with `--session-reuse 1` enabled, `strata-hy4.exe` checks whether the entire `saved_tokens` array (`prev_prompt + prev_generated`) matches `new_tokens[:saved_n]` via a single `memcmp`: - If **any** token differs, it discards the entire cache (`RESUME 0`) and re-prefills from token `0`. - In multi-turn tool-calling chats, re-tokenizing the assistant's previous message text with `tok.encode()` frequently merges BPE pairs differently than token-by-token generation (e.g., generating `108` tokens vs re-encoding `106` tokens). - Whenever those 1–3 trailing tokens in the assistant reply differ, the entire 60,000+ identical prefix before it is thrown away. In our `/api/requests` traces, only a turn where BPE happened to produce the exact same token count hit the cache (`60,730` reused in `0.989 s`); all others missed with `0` reused in `~100 s`. **Suggested fix:** - Pass `"--session-reuse", "1"` by default in `serve/hy4_kind.py` (`engine_args()`). - In `strata-hy4.exe`, find the longest common prefix between `saved_tokens` and `new_tokens` instead of requiring a 100% `memcmp` match on `saved_n`. - (Complementary in `serve/server.py`): for `hy4`, preserve the exact generated `raw_ids` from the previous turn and splice them into the next turn's prompt token IDs when continuing the same conversation prefix. ### Reproduction status Reproduced locally ### Strata version Strata 0.2.19 · strata-hy4 0.1.64 (engine 0.1.65) ### Graphics card, memory and system NVIDIA GeForce RTX 5090 32 GB · 128 GB RAM · Windows 11 ### Model, size and settings Hy4-Flash-SQ-128 · 256K context · --draft dfly2
En el sitio
Enlaces a install, modelos, releases.