Pull requests / #3012

#3012 Community benchmark: Apple M4 Pro (48 GB), Flash-Next Q2_0, 262K context streamed from SSD

open · @ipriyaaanshu · 0 comments · View on GitHub

BenchmarksSetup & installModels & quantsDocumentation

Description

## Summary
Community benchmark report for the Apple Silicon Metal engine on an **Apple M4 Pro with 48 GB unified memory** — a configuration the docs list as untested (the MAC.md measurements stop at a 32 GB M4 mini and an M5 Max).

- Model: Qwen3.8-Flash-Next **Q2_0** (the only size 48 GB fits), **262,144-token context**, experts streamed from SSD (setup's automatic choice + the `.experts` one-request-per-expert copy, `--mmap-experts`, expert cache 64 slots, `--spec 2`).
- Suite: `tools/strata_bench.py` (suite 1), 3 runs per test, engine-own timings, medians with ranges.

## Results (medians)
| Test | Value |
| --- | --- |
| short chat decode | **26.0 tok/s** (23.0-26.8), TTFT 1.35 s |
| 4K fresh prompt | 247.4 tok/s, TTFT 16.3 s |
| 16K fresh prompt | 280.4 tok/s, TTFT 57.3 s |
| 32K fresh prompt | 280.2 tok/s, TTFT 114.4 s |
| agent session | 68.7 % conversation-cache hit, 5 re-read tokens/session, 24.4 tok/s in-session |

Headline decode lands between the docs' 32 GB M4 mini streaming figures (~14 tok/s) and the M5 Max resident figures (108-120) — on the memory-bandwidth-limited part of the curve, as expected for an SSD-streamed pack on an M4.

## Notes for reviewers
- Engine 0.1.45-metal built **from source** (no ready-made Mac binary exists); upstream HEAD did not compile for Metal without local fixes — details are in the report's *Contributor notes* section (missing Metal-compat shims for #2666's new API, and a stale generated `stubs.cpp`).
- No correctness battery was run (speed-only submission); a real agentic workload on the same install (long-document reading) was noticeably faster end-to-end than the dense 27B it replaced, matching the prefill numbers.
- Results-only PR, no engine changes included (the compat patches are described in prose, not committed here, to keep the report self-contained).

Per docs/COMMUNITY_BENCHMARKS.md's submit-a-report workflow.

Related on this site

Quick links to related pages.