Pull requests / #3012
#3012 Community benchmark: Apple M4 Pro (48 GB), Flash-Next Q2_0, 262K context streamed from SSD
open · @ipriyaaanshu · 0 Kommentare · Auf GitHub
BenchmarksSetup & installModels & quantsDocumentation
Beschreibung
## Summary Community benchmark report for the Apple Silicon Metal engine on an **Apple M4 Pro with 48 GB unified memory** — a configuration the docs list as untested (the MAC.md measurements stop at a 32 GB M4 mini and an M5 Max). - Model: Qwen3.8-Flash-Next **Q2_0** (the only size 48 GB fits), **262,144-token context**, experts streamed from SSD (setup's automatic choice + the `.experts` one-request-per-expert copy, `--mmap-experts`, expert cache 64 slots, `--spec 2`). - Suite: `tools/strata_bench.py` (suite 1), 3 runs per test, engine-own timings, medians with ranges. ## Results (medians) | Test | Value | | --- | --- | | short chat decode | **26.0 tok/s** (23.0-26.8), TTFT 1.35 s | | 4K fresh prompt | 247.4 tok/s, TTFT 16.3 s | | 16K fresh prompt | 280.4 tok/s, TTFT 57.3 s | | 32K fresh prompt | 280.2 tok/s, TTFT 114.4 s | | agent session | 68.7 % conversation-cache hit, 5 re-read tokens/session, 24.4 tok/s in-session | Headline decode lands between the docs' 32 GB M4 mini streaming figures (~14 tok/s) and the M5 Max resident figures (108-120) — on the memory-bandwidth-limited part of the curve, as expected for an SSD-streamed pack on an M4. ## Notes for reviewers - Engine 0.1.45-metal built **from source** (no ready-made Mac binary exists); upstream HEAD did not compile for Metal without local fixes — details are in the report's *Contributor notes* section (missing Metal-compat shims for #2666's new API, and a stale generated `stubs.cpp`). - No correctness battery was run (speed-only submission); a real agentic workload on the same install (long-document reading) was noticeably faster end-to-end than the dense 27B it replaced, matching the prefill numbers. - Results-only PR, no engine changes included (the compat patches are described in prose, not committed here, to keep the report self-contained). Per docs/COMMUNITY_BENCHMARKS.md's submit-a-report workflow.
Mehr auf der Site
Links zu Install, Modellen, Releases.