#2844 Community benchmark: Apple M1 Max 64 GB (Metal) — Q2_0 decode 26.7 tok/s, 4K/16K prompts, agent session; 245K needle at 262144 context; Metal OOM note

Beschreibung

Thanks for asking for more Mac data in `docs/MAC.md` — here is the first **Apple M1 Max 64 GB** report (the chip class between your M4 mini and M5 Max measurements). Everything below is from a quiet machine, engine's own timings, `tools/strata_bench.py` suite 1.

## Hardware and software

- **Apple M1 Max** (10-core CPU, 32-core GPU), **64 GB** unified memory, internal SSD, macOS 27.0.1
- Strata project 0.1.44.1, commit 296b59d2, engine **0.1.44.1-metal**, built from source by `setup.sh` on the Mac
- Resident mode: pack mapped via `--mmap-experts`, `STRATA_METAL_STREAM=0`

> Note: the auto-generated report below mislabels the hardware as "CPU only / arm 10 cores / VRAM not read" — that is the Metal engine on the M1 Max GPU; `strata_bench.py`'s hardware probe does not read the GPU on Darwin. The folder name picked up `community-cpu` for the same reason.

## Benchmark report (verbatim, suite 1: short chat / 4K / 16K / agent)

Measured on 2026-10-17 with `tools/strata_bench.py` (suite 1, 3 runs per test, medians with ranges). Speeds are the engine's own timings; first-token times are measured by the client over HTTP. Speed only: no correctness check was run.

### Model and configuration

- Model and quantization: qwen3.8-flash-next-q2_0
- Context: 32768 tokens; parallel slots: 1
- Settings: the install's own configuration, unchanged (metal backend); thinking off for the suite; temperature 0

```text
strata --pack q2_0 --native Qwen3.8-Flash-Next-GSQ-RCO-Q2_0-00001-of-00002.gguf --ple-gguf Qwen3.8-Flash-Next-GSQ-RCO-Q2_0-00002-of-00002.gguf --expert-profile expert-profile.bin --expert-cache 64 --prefill auto --spec 2 --mtp rt --max-context 32768 --mmap-experts
```

Environment of the config: `STRATA_METAL_STREAM=0`

### Results

| Configuration | Actual prompt tokens | Generated tokens | Runs | Prompt tok/s median and range | Decode tok/s median and range | TTFT seconds median and range |
| --- | ---: | ---: | ---: | --- | --- | --- |
| short chat | 46 (43-51) | 100 (100-100) | 9 | not measured | **26.7 (25.4-38.7)** | 1.10 (0.85-1.23) |
| 4K prompt, fresh | 4019 | 24 cap | 3 | **226.0 (222.0-228.8)** | not measured | 17.83 (17.60-18.14) |
| 16K prompt, fresh | 16058 | 24 cap | 3 | **234.1 (228.2-235.3)** | not measured | 68.65 (68.28-70.42) |
| agent session, follow-up turns | 14520 at the last turn | up to 64 | 3 | 184-216 per turn | 28.6 (18.4-31.8) | 13.83 (7.13-16.06) |

Agent session: cache hit 68.7 % over follow-up turns; first turn (nothing cached) first token 3.21 s. For comparison on your M5 Max 128 GB table (Q2_0, same defaults): decode 119.5/108.2 and 4K prompt ~1430 tok/s — the M1 Max lands at ~4.5× slower decode and ~6.3× slower prompt.

## Extra measurements beyond the suite

**Context beyond 32K (your "measured up to 32K" open question).** Ran with `--max-context 262144` (the model's full window), then `tools/needle_bench.py`:

- 64,902 prompt tokens, code word at 90 % depth: **FOUND**, 314 s total
- 246,976 prompt tokens, code word at 90 % depth: **FOUND**, 1116 s total (~221 tok/s effective, prefill+decode; warm file cache)
- The KV cost held: engine RSS 40.3 GB at 32K config → 43.3 GB at 262144 config, swap unchanged. Memory is not the limit on 64 GB; prefill time is (a fresh ~247K prompt is ~19 minutes to first token).

**Quality spot checks (greedy, informal):** 4/4 correct — bat-and-ball (5 with correct algebra), palindrome function, a two-leg average-speed arithmetic problem (63 min), wolf-goat-cabbage (full 7 crossings including taking the goat back). Same answers as a 35B MoE we run locally gets on these.

**Startup:** "decode window … ready in 26759 ms" from engine start; ~65 s from a cold `run-q2_0.sh` to the HTTP API answering.

## A Metal OOM observation (exit code 5, reproducible)

The engine reports "6331 MiB of VRAM free with everything loaded" at startup — ~42 GB wired through Metal for pack+KV+buffers on a 64 GB machine. If another GPU process then wires a large block (in our case a second local inference server loading 17-20 GB of weights), the next Strata request dies with:

```text
mtl: command buffer failed: Insufficient Memory (00000008:kIOGPUCommandBufferCallbackErrorOutOfMemory)
```

and the engine exits with code 5. Two notes that may be useful:

1. After this exit, the server's "the next request restarts the engine" path did not recover: the engine aborted again during restart in the same memory state. A full stop/start of the server (with the other process's memory released first) was needed every time.
2. Our workaround is to release the other server's weights (its `/v1/memory/release`) before each Strata request. Since then: three consecutive requests with a growing conversation, stable 28.8 tok/s, no exits.

Happy to open a PR with the `bench/results/2026-10-17-community-cpu-qwen3-8-flash-next-q2-0/` folder (results.json + engine-log-excerpt.txt included) if that is preferred — the folder name just carries the misdetection above.