#6542 [Engine] Findings on Apple M5 Max 128 GB: Gyro-L and Hy4-Flash for coding agents on v0.2.18 (+ PRs #6538, #6539, #6540)

open · @mnott · 1 评论 · 去 GitHub 看

BenchmarksSetup & installServer & APIAMD / HIPNVIDIA / CUDA

说明

Findings from running Strata v0.2.18 for local coding agents on one Apple Silicon machine, with the PRs that came out of it. This replaces our umbrella issue #6253 (which tracked the v0.2.16 contributions).

## Setup

- Apple M5 Max, 128 GB unified memory, macOS 27.3; Metal maxBufferLength 86.6 GB, recommended working set 115.4 GB.
- Strata v0.2.18 built from source for Metal (ninja + metal-cpp), plus the Hy4 shape engine `strata-hy4` 0.1.64 (release binary and our own build).
- Models: Qwen3.8-Flash-Next Gyro-L (noPLE + PLE Q8_0, rt MTP drafter) and Tencent Hy4-Flash SQ-64 / SQ-96 (+ Engram-Q8, + Vision).
- Clients: Claude Code workers (Anthropic Messages API, streaming, tools) through a local router; several workers at once; one heavy model loaded at a time.

## PRs

- #6538 — SSD conversation tier: fingerprint the resolved MTP files, keep the tier across restarts (rebased on v0.2.18; supersedes #6305 and #6439, fixes #6281).
- #6539 — Hy4-Flash (Metal): resume at the longest common prefix, several conversation slots, SSD tier kept across restarts.
- #6540 — Hy4-Flash: SQ-96 starts on 128 GB Macs, `--info` no longer crashes, session reuse on by default, `reasoning_effort: max` in configs (stacked on #6539).

## Findings, Qwen3.8-Flash-Next Gyro-L on v0.2.18

- Cold prompt reads 1,057-1,136 tok/s (8k-52k tokens; v0.2.16 on the same machine: 1,011 at 10k, 858 at 75k), decode 70-117 tok/s with drafts.
- With #6538: prefix reuse 51k of 52k on agent turns; 110 conversations adopted from the SSD tier after a restart.
- It remains our coding-agent model: on two real repository tasks it passed both (reviewed 4/5 each).

## Findings, Hy4-Flash

- **Packs on 128 GB:** SQ-64 (62 GB) maps whole: ~69 GB Metal allocated, prompt reads ~1,070 tok/s (46k in 43 s), decode 65-86 tok/s with DFly2. SQ-96 (89.6 GB) and SQ-128 exceed maxBufferLength and run in cold-expert mode: prompt reads ~260 tok/s, memory ~90%, CPU peaks from expert reads. SQ-96 did not start at all without `STRATA_METAL_EXP_CACHE_GB` (fixed in #6540). The setup recommends SQ-128 on this machine ("fits in memory"); in practice SQ-64 is the one to run on 128 GB.
- **Prompt reuse:** off by default and, when on, exact-prefix only, so agent loops re-read the whole prompt every turn (44 s for 46k; ~100 s per turn on SQ-96). With #6539: 0.3 s for a repeated 46k prompt, 1.1-1.3 s for agent-style turns, two concurrent agents keep their prefixes (1-12 s prompt reads at 39-60k).
- **Reasoning levels:** `no_think` answers clean code but answers a tool-call probe in prose ("I don't have access…"); `high` calls the tool. At `high` one agent turn reasoned 18k tokens / 257 s; a `reasoning_budget_tokens` of 4096 in the run config keeps turns short and keeps rollbacks under the 8192-token ring slack (beyond it the engine recomputes fully). At `max` the model continued reasoning in the answer channel after the budget (13-18k-token answers).
- **CPU:** ~1 core during decode (host build ~457 ms per plain step vs ~63 ms GPU; plain steps are rare with DFly2), multi-threaded peaks during load/prefill and in cold-expert mode.
- **Quality on our agent tasks** (Claude Code, same two tasks as Gyro-L, reviewed by a separate model): task 1 passed 2/2 (4/5, same as Gyro-L, ~3x slower); task 2 failed 3/3 (twice at `high`, once at `max`) with the same mistake in its own test fixture (a temp-dir fixture the guard under test exempts), correctly diagnosed at `max` but not fixed. Gyro-L passed both.

## Also noted

- `tools/test_setup_hy4.py::MacHy4Flash::test_mac_install_plan_defaults_to_128k_and_unified_fit` fails on a clean v0.2.18 checkout on macOS (a data dir under `/var/folders` is treated as a system folder).
- The CUDA/HIP/SYCL Hy4 serve loops have the same exact-prefix reuse check as the Metal one; #6539 changes Metal only.

本站相关内容

相关页面的快捷入口。