#6539 [Engine] Hy4-Flash (Metal): resume at the longest common prefix, several conversation slots, SSD tier kept across restarts
open · @mnott · 0 commentaires · Sur GitHub
BenchmarksServer & APIAMD / HIPNVIDIA / CUDAModels & quantsWindows
Description
## Problem and result On Hy4-Flash (Metal), coding agents (Claude Code, other agent loops with 20-60k-token prompts) re-read the whole prompt on every turn. The serve loop reuses KV only when the entire previous resident sequence (prompt + generated tokens) is an exact prefix of the new prompt (`hy4_metal_serve.h`). In an agent loop that never holds: the chat template re-renders earlier assistant turns (thinking dropped) and the last user message changes. Measured on SQ-64, Apple M5 Max 128 GB: a 46k-token prompt takes 44 s every turn; a 25k agent prompt ~100 s per turn on SQ-96. The engine also keeps only one conversation, so two agents sharing the engine evict each other completely. This PR (Metal path only), three commits: 1. **Resume at the longest common prefix.** With `--session-reuse 1`, the engine keeps the KV of the longest common prefix with the previous sequence and recomputes only from the first differing token (bounded by the sliding-window ring slack; a rollback beyond it recomputes fully, by design). Indexer, Engram prefetch and DFly2 drafter state are rolled back consistently. 2. **Several conversation slots** (`--conversation-slots N`, default 4 with reuse on; `--conversation-cache-mib`, `--conversation-min-tokens`): each slot keeps its own KV set; a request takes the slot with the longest common prefix, else the least recently used one. 3. **SSD conversation tier, kept across restarts**, mirroring the main engine's flags (`--conversation-spill-dir`, `--conversation-spill-mib`, `--conversation-spill-persist`): a slot evicted from RAM is written in the background with its token ids; a request whose best prefix is on SSD loads it back instead of recomputing; with persist, the folder (named by the model fingerprint) survives a restart. | measured (SQ-64, M5 Max 128 GB) | before | after | |---|---|---| | same 46k prompt again | 44 s, 0 cached | 0.3 s, 46,127 cached | | agent-style turn, ~17k, history and last message changed | 20-22 s | 1.1-1.3 s, ~17.1k cached | | two conversations A,B,A,B (2 slots) | 10-12 s each | 0.4 s on return, ~11.5k cached each | | slot evicted to SSD / reloaded | - | 650 MiB written in 112 ms (background) / reloaded in 63 ms | | restart with persist | - | conversations adopted at start | | real agent run: two Claude Code workers sharing the engine | - | prompt reads of 39-60k tokens in 1-12 s while the engine switched between them | ## Related discussion none (a summary issue with our Hy4 findings on Apple Silicon is linked from it). ## Approach - The reuse decision is a pure function (longest common prefix of the resident sequence and the new prompt, minus a floor so the last prompt token is always recomputed for its logits); the existing `RESUME <n>` protocol line reports it, so serve's `cached_tokens` needs no change. - Slots: one KV set per slot (`Runner::alloc_kv_set`), allocated on first use; the slot picker requires a minimum shared prefix (`--conversation-min-tokens`, default 1024) before preferring a slot over the LRU one, so an 18-token template overlap does not make B overwrite A. - SSD tier: a background writer, a fingerprint of the model file, index files beside the KV files, adoption at start only for a matching fingerprint (same rules as the main engine's tier). - Compatibility: with `--session-reuse 0` (or absent) the engine behaves exactly as before; no change to the model files or the protocol. ## Validation **Passed:** - Built with `engine/src/hy4flash/metal/build_mac.sh run` on v0.2.18 + these commits, macOS 27.3, Apple clang, M5 Max 128 GB. - Greedy (temperature 0, no_think) output with reuse equals a fresh run on 3 prompts, DFly2 on; interleaved A,B,A,B continuations equal fresh runs. - The measurements in the table above (serve on port 8039, two-turn prefix test and two concurrent Claude Code workers through an Anthropic-API router). - `python3 -m pytest serve/test_server.py serve/test_hy4_dialect.py serve/test_effort_paths.py serve/test_hy4_serve_e2e.py -q` (with the follow-up PR on top): 498 passed, 734 subtests. **Failed:** - none observed. **Not run:** - CUDA, HIP and SYCL Hy4 engines (they have the same exact-prefix check in their serve loops and would need the same change); Windows. - Decode speed was not benchmarked separately; agent turns showed 46-56 tok/s with DFly2, in line with before. ## Limits - A rollback larger than the sliding-window ring slack (8192 tokens) falls back to a full recompute. Long discarded thinking (e.g. >8k thinking tokens that the next prompt drops) therefore costs one full re-read; a thinking budget below the slack (`reasoning_budget_tokens`) avoids it. - Each slot's KV set is sized for the full context (3.16 GB at 131072), so slots cost memory up front once used. - Images: requests with pictures do not reuse KV (unchanged behaviour).
Sur le site
Liens install, modèles, releases.