#2656 Community benchmark: Apple M1 Max (24-core GPU, 32 GB), Qwen3.8-Flash-Next Q2_0 (Metal, streaming)

Beschreibung

# Community benchmark on Apple M1 Max (24-core GPU, Metal)

Measured on 2026-10-16 with `tools/strata_bench.py` (suite 1, 3 runs per test, medians with ranges). Speeds are the engine's own timings; first-token times are measured by the client over HTTP. Speed only: no correctness check was run.

> Note: `strata_bench.py` probes `nvidia-smi`/`rocm-smi` for the GPU and, finding neither on macOS, auto-labelled this run "CPU only". That is a detection gap, not the actual device: the run used the **Apple M1 Max 24-core GPU via Metal** (engine build `0.1.44.1-metal`, `--mmap-experts` streaming). The hardware fields below are filled in by hand; all timing numbers are the tool's unmodified output.

## Hardware and software

- GPU: Apple M1 Max, 24-core GPU, Metal 4. No dedicated VRAM — 32 GiB unified memory shared with the system. On 32 GiB the Q2_0 pack (~37.6 GB mapped) does not fit Metal's resident share (~26 GiB), so the experts stream from the SSD automatically (`--mmap-experts`); the OS file cache serves them as the GPU's expert cache fills.
- CPU: Apple M1 Max (10 cores: 8 performance + 2 efficiency); installed RAM: 32.0 GiB unified; storage: Apple NVMe SSD (AP0512R, 500 GB, TRIM supported).
- OS: macOS 26.5.1 (Darwin 25.5.0, build 25F80); Metal 4; GPU driver: n/a (Metal, built into the OS).
- Strata: project 0.1.44.1, commit 296b59d2; engine 0.1.44.1-metal (built from source).
- Background workloads and power limits: desktop idle, no other GPU workload during the run; on AC power.

## Model and configuration

- Model and quantization: qwen3.8-flash-next-q2_0
- Context: 32768 tokens; parallel slots: 1
- Settings: the install's own configuration, unchanged; thinking off for the suite; temperature 0

## Method

Run by `python tools/strata_bench.py --url http://127.0.0.1:8080` against the HTTP API of a Strata server (already running); model loading is not timed. One warm-up chat and a 1K prompt are discarded first. Every prompt starts with a unique tag, so the conversation cache cannot help the fresh-prompt tests.

- short chat: the three questions of `tools/ab_engine.py`, up to 100 new tokens, 3 runs
- 4K / 16K prompts: generated word lists sized by a calibration request, 24 new tokens, 3 fresh prompts each
- agent session: system prompt and four tools, then 5 turns that each add about 1500 words of a project file and continue from the model's own earlier answers (the way agent clients resend a conversation), 64 new tokens per turn, 3 sessions; the cache numbers come from the answer's usage (`cached_tokens`) and timings (`prompt_n`)
- concurrency (when run): N requests sent at the same moment, total tokens over the wall time of the burst, latency over all requests

## Results

| Configuration | Actual prompt tokens | Reused tokens | Generated tokens | Runs | Prompt tok/s median and range | Decode tok/s median and range | TTFT seconds median and range |
| --- | ---: | ---: | ---: | ---: | --- | --- | --- |
| short chat | 46 (43-51) | 0 | 100 (100-100) | 9 | not measured | 14.7 (13.2-15.1) | 2.95 (2.42-3.19) |
| 4K prompt, fresh | 4019 (4019-4019) | 0 (0-0) | 24 cap | 3 | 289.4 (278.1-290.0) | not measured | 13.94 (13.91-14.51) |
| 16K prompt, fresh | 16058 (16058-16058) | 0 (0-0) | 24 cap | 3 | 287.1 (285.5-287.2) | not measured | 55.99 (55.98-56.31) |
| agent session, follow-up turns | 14514 (14509-14532) at the last turn | see below | up to 64 | 3 | see below | 16.6 (14.1-19.6) | 11.95 (6.92-14.74) |

Ranges are the lowest and highest of the runs; `Runs` counts requests that returned a token.

### Agent session (conversation cache)

Cache hit over the follow-up turns: 68.7 % (68.7-68.7); earlier-turn tokens read again per session: 5 (5-5) (earlier turns that the cache did not hold; the 13730 (13730-13730) new tokens of the files are read once); first turn (nothing cached) first token: 3.33 s (3.32-3.34).

| Turn (run 1) | Prompt tokens | Reused | Read | Prompt tok/s | First token (s) | Decode tok/s |
| ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| 0 | 609 | 0 | 609 | 186.2 | 3.33 | 14.1 |
| 1 | 3751 | 628 | 3123 | 237.0 | 13.238 | 14.1 |
| 2 | 5284 | 3814 | 1470 | 210.4 | 7.043 | 15.3 |
| 3 | 8728 | 5347 | 3381 | 232.3 | 14.616 | 16.5 |
| 4 | 11616 | 8754 | 2862 | 240.8 | 11.946 | 18.5 |
| 5 | 14532 | 11633 | 2899 | 240.0 | 12.14 | 19.6 |

Per-request data: `results.json`. Engine log lines from the start: `engine-log-excerpt.txt` (when this run started the server).

## Correctness and limitations

Not measured: this is a speed suite. The conversation uses the model's own replies, so token counts differ a little between runs. Machine details were filled in by hand (the tool cannot read a Metal GPU); background load and power limits are as noted above. Total time of the suite: 540 s.

The 32K prompt test (`--long`) was skipped: the install's context is 32768 tokens, so a ~32K prompt leaves no room for the generation, and the tool skips it. 16K is the longest prompt measured here. The suite was run twice with closely matching numbers (chat decode 14.7 / 14.9 tok/s, 4K prompt 289 / 286 tok/s, 16K prompt 287 / 286 tok/s), so the medians above are reproducible.

For context, Strata's own `docs/MAC.md` measures Q2_0 on a 32 GB **M4 Mac mini** (streaming) up to a 4K prompt: ~14 tok/s decode and ~94 tok/s on the 4K prompt. This M1 Max 32 GB run lands at a similar ~15 tok/s decode but ~287 tok/s fresh-prompt throughput — about 3x the M4 mini's prompt speed (the 24-core GPU and wider memory bandwidth; decode stays SSD-bound, so it matches). 16K and the agent session are beyond what the official M4 mini report covers. A previously unreported Apple Silicon configuration.