#6539 [Engine] Hy4-Flash (Metal): resume at the longest common prefix, several conversation slots, SSD tier kept across restarts

open · @mnott · 0 comentarios · En GitHub

BenchmarksServer & APIAMD / HIPNVIDIA / CUDAModels & quantsWindows

Descripción

## Problem and result

On Hy4-Flash (Metal), coding agents (Claude Code, other agent loops with 20-60k-token prompts) re-read the whole prompt on every turn. The serve loop reuses KV only when the entire previous resident sequence (prompt + generated tokens) is an exact prefix of the new prompt (`hy4_metal_serve.h`). In an agent loop that never holds: the chat template re-renders earlier assistant turns (thinking dropped) and the last user message changes. Measured on SQ-64, Apple M5 Max 128 GB: a 46k-token prompt takes 44 s every turn; a 25k agent prompt ~100 s per turn on SQ-96. The engine also keeps only one conversation, so two agents sharing the engine evict each other completely.

This PR (Metal path only), three commits:
1. **Resume at the longest common prefix.** With `--session-reuse 1`, the engine keeps the KV of the longest common prefix with the previous sequence and recomputes only from the first differing token (bounded by the sliding-window ring slack; a rollback beyond it recomputes fully, by design). Indexer, Engram prefetch and DFly2 drafter state are rolled back consistently.
2. **Several conversation slots** (`--conversation-slots N`, default 4 with reuse on; `--conversation-cache-mib`, `--conversation-min-tokens`): each slot keeps its own KV set; a request takes the slot with the longest common prefix, else the least recently used one.
3. **SSD conversation tier, kept across restarts**, mirroring the main engine's flags (`--conversation-spill-dir`, `--conversation-spill-mib`, `--conversation-spill-persist`): a slot evicted from RAM is written in the background with its token ids; a request whose best prefix is on SSD loads it back instead of recomputing; with persist, the folder (named by the model fingerprint) survives a restart.

| measured (SQ-64, M5 Max 128 GB) | before | after |
|---|---|---|
| same 46k prompt again | 44 s, 0 cached | 0.3 s, 46,127 cached |
| agent-style turn, ~17k, history and last message changed | 20-22 s | 1.1-1.3 s, ~17.1k cached |
| two conversations A,B,A,B (2 slots) | 10-12 s each | 0.4 s on return, ~11.5k cached each |
| slot evicted to SSD / reloaded | - | 650 MiB written in 112 ms (background) / reloaded in 63 ms |
| restart with persist | - | conversations adopted at start |
| real agent run: two Claude Code workers sharing the engine | - | prompt reads of 39-60k tokens in 1-12 s while the engine switched between them |

## Related discussion

none (a summary issue with our Hy4 findings on Apple Silicon is linked from it).

## Approach

- The reuse decision is a pure function (longest common prefix of the resident sequence and the new prompt, minus a floor so the last prompt token is always recomputed for its logits); the existing `RESUME <n>` protocol line reports it, so serve's `cached_tokens` needs no change.
- Slots: one KV set per slot (`Runner::alloc_kv_set`), allocated on first use; the slot picker requires a minimum shared prefix (`--conversation-min-tokens`, default 1024) before preferring a slot over the LRU one, so an 18-token template overlap does not make B overwrite A.
- SSD tier: a background writer, a fingerprint of the model file, index files beside the KV files, adoption at start only for a matching fingerprint (same rules as the main engine's tier).
- Compatibility: with `--session-reuse 0` (or absent) the engine behaves exactly as before; no change to the model files or the protocol.

## Validation

**Passed:**
- Built with `engine/src/hy4flash/metal/build_mac.sh run` on v0.2.18 + these commits, macOS 27.3, Apple clang, M5 Max 128 GB.
- Greedy (temperature 0, no_think) output with reuse equals a fresh run on 3 prompts, DFly2 on; interleaved A,B,A,B continuations equal fresh runs.
- The measurements in the table above (serve on port 8039, two-turn prefix test and two concurrent Claude Code workers through an Anthropic-API router).
- `python3 -m pytest serve/test_server.py serve/test_hy4_dialect.py serve/test_effort_paths.py serve/test_hy4_serve_e2e.py -q` (with the follow-up PR on top): 498 passed, 734 subtests.

**Failed:**
- none observed.

**Not run:**
- CUDA, HIP and SYCL Hy4 engines (they have the same exact-prefix check in their serve loops and would need the same change); Windows.
- Decode speed was not benchmarked separately; agent turns showed 46-56 tok/s with DFly2, in line with before.

## Limits

- A rollback larger than the sliding-window ring slack (8192 tokens) falls back to a full recompute. Long discarded thinking (e.g. >8k thinking tokens that the next prompt drops) therefore costs one full re-read; a thinking budget below the slack (`reasoning_budget_tokens`) avoids it.
- Each slot's KV set is sized for the full context (3.16 GB at 131072), so slots cost memory up front once used.
- Images: requests with pictures do not reuse KV (unchanged behaviour).

En el sitio

Enlaces a install, modelos, releases.