#3571 [Benchmark] Apple M4 Max 64 GB / Qwen3.8-Flash-Next Q2_0 / strata-qwen38-thinking-v1-draft.1 (complete)

Description

Results only, no engine change, no defaults touched. Adds one report folder: an
Apple Silicon data point the repository does not have yet.

## Problem and result

`docs/MAC.md` publishes numbers for an M5 Max at 128 GB, an M4 mini at 32 GB and an M1 Max at 34 GB, and states that other chips are untested. `bench/results/` contains no Apple Silicon row at all. An M4 Max at 64 GB is the size at which the table in that document recommends Q2_0 as the largest pack that stays resident (it puts IQ3_XXS at 74 GB), so it fills the gap between the M4 mini and the M5 Max.

This adds a standard run on that machine. **Status: complete** — 21 of 21 required requests finished, 0 failed, 0 skipped, 12 of 12 sanity checks.

| Case | Prompt tok/s | Decode tok/s | TTFT s | First answer s | Total s |
| --- | ---: | ---: | ---: | ---: | ---: |
| short | 183.5 (134.9–203.6) n=3 | 72.4 (72.1–72.6) n=3 | 0.44 | 1.19 | 7.5 |
| prompt-4k | 692.4 (647.1–693.3) n=3 | 69.2 (68.5–69.8) n=3 | 5.93 | 8.21 | 12.0 |
| prompt-16k | 683.8 (682.8–684.5) n=3 | 66.0 (65.7–66.6) n=3 | 23.99 | 28.65 | 31.7 |

Medians in tokens/s, n=3. Cells are median (min–max); full table, phase counts and the sanity-check detail are in the folder README.

## Related discussion

- #3561 — the release ships no `COMMIT` file, which made the first attempt at this run `nonconforming`. Reported separately because it is a packaging bug, not a result. This PR is the result it blocked.
- None required otherwise; no scope change, no new default.

## Approach

Ran the standard suite from the repository checkout so the report lands in `bench/results/` and the recorded command carries a repo-relative `--out` rather than a local absolute path:

```
strata bench run --profile macstudio --track default \
  --track-notes "Strata 0.2.1 release, engine 0.1.46 macos-arm64-metal. Setup defaults: model q2_0, context 32k, vision gpu, tune skip. Model files on an NVMe SSD; no other GPU or CPU-heavy workloads." \
  --model-revision "ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF@ed59f92082b1e93c0e96d60a8b11aab089b52f09" \
  --source-revision "Qwen/Qwen3.8-Flash-Next@de4b8e4d43b917e7706784d8bb445c9af86a3540" \
  --tokenizer-revision "generated by tools/strata_tokenizer.py from the Q2_0 GGUF (setup.py step 6); no upstream tokenizer revision applies" \
  --quantization "Q2_0" \
  --background "Mac Studio M4 Max, 64 GB unified memory, macOS 27.0.1 (26A434), AC power, no power limit set; no other GPU/CPU-heavy workloads running" \
  --out bench/results/2026-10-21-community-m4-max-q2_0
```

Track `default` throughout: no `strata tune`, no `--set`, no rebuilt engine. Setup defaults only. The profile was created with `strata profile create macstudio --from-config ~/.local/share/strata/install/strata-q2_0.json` and validated before the run.

`bench/results/COMMUNITY.md` is untouched — the index is the maintainers' to update.

## Validation

**Passed:**

- `strata bench run --profile macstudio --track default ...` (command above) — Strata 0.2.1, commit `90767252f420e25223c739d604145ad15911d48b`, engine 0.1.46 `macos-arm64-metal`; macOS 27.0.1 (26A434) on Apple M4 Max, 64 GB unified memory. Observed: `Status: complete. 21 of 21 required requests finished, 0 failed, 0 skipped`, 12 of 12 sanity checks (instruction 4/4, retrieval 4/4, tool calls 4/4). Evidence: `results.json`, `outputs.jsonl` (every request with raw usage, engine timings and client offsets) and `provenance/` in this folder.
- Reproducibility across three complete runs on this machine, each from a freshly loaded engine, same release and configuration: decode 72.4 / 72.9 / 72.9 tok/s and prompt 683.8 / 694.2 / 696.3 tok/s. Decode varies under 1%, prompt under 2%. Table in the folder README.
- Sanity checks I verified by reading the outputs rather than trusting the summary: the four retrieval and four tool-call cases passed on expected content.

**Failed:**

- The first attempt at this run (2026-10-20) ended `Status: nonconforming` with "the Strata commit is unknown (not a git checkout and no COMMIT file)". No release artifact ships `COMMIT`; the code itself says a release must carry one. Worked around by writing the commit the annotated tag `v0.2.1` dereferences to into the installed terminal package, then re-running from scratch. Reported as #3561. No measurement from the failed attempt is included in this PR.
- No other failure observed. No request failed in any of the three complete runs.

**Not run:**

- **Optional concurrency stage.** `--optional-concurrency` was not passed, so `concurrency_rounds` is empty. Four concurrent short requests would need a second check of server capacity on this machine; this report covers single-request throughput only.
- **`tools/needle_bench.py` long-context recall.** Not run. The protocol's 12 sanity cases already include 4 retrieval and 4 tool-call checks, which are this submission's correctness gate; the needle script targets recall at context lengths beyond the 32768 this profile is configured for. Untested: recall degradation at 64K/128K on this machine.
- **Source build, and CUDA / HIP / SYCL engines.** Not applicable — results-only, no code or engine change. Nothing was compiled.
- **`strata tune`.** Not run, deliberately: tuning would move the run off track `default`.
- **Any second GPU.** This is a single-GPU unified-memory Mac; multi-GPU behaviour is untested here.

## Limits

- **The protocol is a draft.** `strata-qwen38-thinking-v1-draft.1` is not activated, and `protocol.model_card_revision` is null for every submitter until the pilot runs. This report is not an approved standard result and should not be read as one.
- **The Strata commit in this report was placed by hand**, as described above. The engine binary is the unmodified release package, so `build_kind: release` is accurate, but a fresh install of v0.2.1 will not find that file.
- **The tokenizer has no upstream revision.** The pack's `vocab.json`, `merges.txt` and `token_type.json` are generated from the GGUF by `tools/strata_tokenizer.py`, not copied from Hugging Face — verified, not assumed: all three differ byte for byte from `Qwen/Qwen3.8-Flash-Next@de4b8e4d`, and `token_type.json` does not exist upstream. `--tokenizer-revision` records the derivation rather than naming a revision that does not exist.
- **Three nulls are Apple Silicon gaps**, not skipped measurements: `hardware/storage`, `software/drivers` and `gpus[0].power_limit_w`. The runner identifies the drive and the GPU driver through Linux and Windows tooling that does not exist here. The model files did sit on an internal NVMe SSD.
- **32768-token context only.** No 128K sweep, so this row is not comparable with community rows that report 4K / 32K / 128K.
- **5 of 9 performance requests were truncated** at the 512-token cap (visible in the phase table). Truncation stays inside the speed summaries by protocol; no sanity case failed. The `short` case also shows a wide prompt-throughput spread (134.9–203.6) from a short prompt whose first request pays first-use cost.
- **`engine.log` is not in this folder.** The run produced an empty one and `*.log` is gitignored in this repository, so there is no engine log to attach.