#6464 [Benchmark] Strix Halo 64 GB unified memory (Ryzen AI Max+ 395)

open · @ryougishiki00 · 0 comentários · No GitHub

BenchmarksServer & APIAMD / HIPNVIDIA / CUDAModels & quantsDocumentationWindowsLinux

Descrição

## Problem and result

`docs/COMMUNITY_BENCHMARKS.md` has one Strix Halo entry (128 GB, 70 W) and the maintainer numbers are 128 GB too. On those
machines all 24,576 experts fit the GPU-visible pool. On a 64 GB Strix Halo the pool is the same memory the OS lives in, so
only ~2.6k–4.7k expert-cache slots fit and the choice of BIOS carve-out, prompt chunk and drafter starts to matter. There
was no public 64 GB data point to plan against.

This adds three things, docs only — no code, no defaults, no behaviour change:

- `docs/bench-results/2026-11-21-community-strix-halo-64gb/` — the report: what the 64 GB machine measures, the ablations
  behind each number (carve-out, fast config, `amd_iommu=off`, expert-cache, prefill chunk, both drafters), and what did
  not help;
- `docs/bench-results/2026-11-21-standard-8060s-7d853232/` — the standard-protocol run that the report links to, byte for
  byte as `strata bench run` wrote it (one added `NOTES.md` explains the `-unknown-` → `-8060s-` directory rename);
- one line in `docs/COMMUNITY_BENCHMARKS.md`.

Headline, official ISTA-DASLab IQ3_XXS + locally packed MTP drafter, 512 MiB carve-out, 61.94 GiB `MemTotal`:

- standard protocol `strata-qwen38-thinking-v1-draft.1`: complete, 21 of 21 required requests, 0 failed, 0 skipped,
  12 of 12 sanity checks, 4 min 45 s after the model loaded, `track: custom`;
  prompt 939.9 tok/s at 4K and 1,180.5 at 16K, decode 34.0–42.4 tok/s, TTFT 0.81 / 4.39 / 13.92 s;
- greedy sweep (thinking off, fresh salt per request): 1,058–1,173 tok/s prompt at 110K–260K tokens, the whole 256K window
  usable (260,513-token prompt at 1,058 tok/s);
- the SSD file tier is never touched — the 39.97 GiB expert arena stays in pinned host memory, so a small GPU cache is
  nearly free on decode (hit 81% vs 88% measures the same, ±0.5 tok/s);
- prompt throughput reaches the 128 GB numbers; decode is ~10 % lower on code-like content (50–52 vs 57.8 tok/s, both
  greedy) and the cause is not established here.

What I hope it saves other 64 GB owners: take the smallest BIOS carve-out (512 MiB here → 61.94 GiB `MemTotal`; 32/48 GiB
starve the OS and push experts back into the unbuffered file tier), apply `docs/STRIX_HALO.md` §5 (+62–64 % prompt), keep
at least ~1,600 cache slots because the prompt path borrows them, and fetch/convert the MTP head (it is not in the GSQ-RCO
pack) and name it on the command line — without `--mtp` the engine drafts nothing at all.

## Related discussion

none — benchmark report, which `CONTRIBUTING.md` allows straight to a PR.

## Approach

Results only. The `strata bench run` output is copied unmodified (the only added file is `NOTES.md`); the community report
follows the format in `docs/COMMUNITY_BENCHMARKS.md`, states for every number which weights, which drafter and which flags
ran, and keeps the standard-protocol rows separate from the hand-run sweep rows so they are not compared across methods.
No engine, GUI, terminal or documentation outside these three paths changes.

## Validation

**Passed:**
- `strata bench run --profile launch-iq3_xxs-256k --track custom --model-revision ed59f920… --source-revision de4b8e4d…`
  on v0.2.15 / engine 0.1.60 `linux-x86_64-hip-halo`, Ubuntu 26.04.1, gfx1151 (Radeon 8060S, 64 GB, 512 MiB carve-out):
  complete, 21 of 21 required, 0 failed, 0 skipped, 12 of 12 sanity (instruction 4/4, retrieval 4/4, tool calls 4/4),
  4 min 45 s after load — `docs/bench-results/2026-11-21-standard-8060s-7d853232/results.json`, engine log in the same
  folder.
- Cold start: the run's own README records `The server had answered no request before this run (a freshly loaded engine)`
  and `config.json` lists the effective engine options (`--kv int8`, `--expert-cache 2000`, `--prefill 8192`, `--mtp`,
  `--mtp-window 8192`, `--spec-min-p 0.5`, `--max-context 262144`).
- Hand-run sweep (greedy, thinking off, fresh salt per request, `/v1/chat/completions`): prompt 110,454 tokens
  1,122.8–1,173.4 tok/s; 177,198 tokens 1,081.9; 232,930 / 247,867 / 260,513 tokens 1,058–1,078 tok/s, all above
  `MemAvailable` floor 2,560 MiB (lowest seen 2,836 MiB).
- Documentation links in the two added READMEs resolve inside `docs/bench-results/`.

**Failed:** none observed in the runs reported here.

**Not run:**
- CUDA, Metal and SYCL backends — no such hardware here;
- the optional concurrency stage of the protocol (`--optional-concurrency` was not selected, so `optional_duration_s` is
  null by design);
- the Windows numbers quoted in the report are from the same machine in an earlier session, not re-run on v0.2.15;
- model file hashing beyond what the runner did itself.

## Limits

- The 64 GB pool holds ~2.6k–4.7k expert slots instead of 24,576, and decode lands ~10 % below the 128 GB entry on code-like
  content. The obvious candidate (expert-cache coverage, 61 % vs 100 %) is not established: raising the cache from 2,649 to
  5,885 slots moved decode by ~4 %, and the two machines differ in engine version, quantization and prompt set.
- Drafter choice moves decode more than cache size does: MTP accepts 60 % (prose) / 85 % (code) and is 17 % faster than
  DFlash2 on prose while tying it on code, so every table names the drafter instead of mixing them.
- One observation while preparing this: `strata serve --profile` re-resolves `drafting` and replaced an explicitly chosen
  MTP drafter with DFlash2 when `--mtp <folder>` did not contain `experts.bin`, silently (a note only, in the engine log).
  The run committed here loads the MTP layer correctly and the log line `strata mtp: draft layer loaded` is in
  `engine.log`; the earlier, lower run was discarded.
- All numbers are one machine (one laptop, one BIOS version, one NVMe); the carve-out and `amd_iommu=off` findings may not
  transfer to other 64 GB Strix Halo models.

No site

Links install, modelos, releases.