#6410 [Engine] Scope an opt-in compact two-slot mode for resident Metal
open · @ain-soph · 0 コメント · GitHub で見る
BenchmarksAMD / HIPModels & quants
本文
### Area [Engine] ### What do you want to do? Could we scope an opt-in Metal mode for two resident Flash-Next batch slots that uses two physical sequence states? On the fork baseline, two slots allocate three states: the two decode slots and a separate main state for prompt reads. The prototype borrows an idle slot for prefill and restore. When the matching prefix is in the other idle slot, it can exchange logical slot ownership without copying the KV or discarding the other prefix. The proposed scope is Metal, resident Flash-Next, exactly two slots, and the existing verifier with per-slot MTP disabled. The default would keep the existing allocation. This matters on Apple Silicon when a long-context chat server shares unified memory with another GPU service. On an M5 Ultra (96 GB), AtomicChat Flash-Next AD-4.27, INT8 KV, 262,144 tokens per slot, the fork measured 70,962 → 67,695 MiB peak wired increment. Four greedy short answers matched byte for byte. Short two-stream throughput was 151.9 → 151.2 tokens/s combined. Two ~35K continuations each reused their prefix, including an observed logical slot exchange. A separate Navin AD-4.27 run on the same machine retrieved both markers from two ~257K / ~260K conversations, then issued both continuations in reverse order. They reused 256,676 and 259,622 tokens, processing 25 new tokens each. These are measurements of the fork prototype against its own baseline, not results against current upstream. The prototype is public in ain-soph/Strata at 4cf59801f142f6d6e8e6a71a192a69bc63465abe. Before a PR I would port the changes to current main and test both the existing default and the opt-in path, including prefix restore, cancellations, and two active streams. Would this scope be useful, and are there other Metal slot cases you would want covered first? ### What do you do today instead? Use one slot or reserve enough memory for the third state. The fork currently uses the opt-in mode on one M5 Ultra.
関連リンク
インストール・モデル・リリースへのサイト内リンク。