Pull requests / #3581

#3581 [Benchmark] Radeon VII 16 GB (gfx906) - standard-v1 (complete) + --pool-workers sweep (release v0.2.1.4, engine 0.1.46.3)

open · @Kaidora · 0 comments · View on GitHub

BenchmarksServer & APIMulti-GPUAMD / HIPModels & quantsDocumentation

Description

## Problem and result

A complete run of the draft standard protocol on one AMD Radeon VII 16 GB
(gfx906), Qwen3.8-Flash-Next Q2_0 with MTP, 65536-token context, release v0.2.1.4 (engine 0.1.46.3) built from source
for gfx906 with no local patches.

`strata bench run --suite standard-v1`, started from a freshly loaded engine (the report notes no earlier requests),
status **complete**: 21 of 21 required requests, 0 failed, 0 skipped, **12 of 12 sanity checks passed** (instruction
4/4, retrieval 4/4, tool calls 4/4). Time after the model loaded: 5 min 37 s.

| case | prompt tok/s | decode tok/s | first token | total |
| --- | ---: | ---: | ---: | ---: |
| short | 65.1 (65.0-67.9) | 50.0 (43.9-53.0) | 1.22 s | 9.6 s |
| prompt-4k | 484.5 (483.6-485.8) | 45.7 (44.5-51.0) | 8.49 s | 19.6 s |
| prompt-16k | 524.2 (514.2-527.1) | 47.8 (45.5-49.9) | 31.32 s | 42.5 s |

The folder also carries a ten-run supplementary sweep of the community guide's quick suite (`pool-workers-sweep/`).
It is attached because the protocol fixes its prompts, thinking and sampling for cross-machine comparison, while the
one open question on this machine needs a single setting varied: the engine's default for `--pool-workers` is 15 workers
on this 16-core CPU, and 10 or 6 decode 11-23 % faster (three repeats of the default arm at each context put the effect
at 3.8-5.7x the replicate spread at 65536 and 1.9-2.2x at 262144, permutation p <= 0.0016). Those ten runs are **not**
standard-protocol results and the report says so.

## Related discussion

none

## Approach

Results-only: one new folder under `bench/results/` with the standard run's `README.md`, `results.json`,
`outputs.jsonl`, `config.json`, `provenance/` (chat template, corpus build and manifest), plus `pool-workers-sweep/`
with ten `results.json` files and its own README. No engine code changes.

The protocol is a draft (`strata-qwen38-thinking-v1-draft.1`); this report records its draft status and does not claim
standard or approved status. The run is `track: custom` because the configuration is tuned (`--expert-cache auto`,
`--kv int8 --kv-resident 32768`, `--spec 4 --spec-min-p 0.5`, a local MTP directory, `--pool-workers 6`).

## Validation

**Passed:**
- Build: `cmake -S <checkout at tag v0.2.1.4 (engine 0.1.46.3)> -B build-906 -G Ninja -DSTRATA_HIP_GFX906=ON
  -DCMAKE_HIP_ARCHITECTURES=gfx906 -DSTRATA_GGML_DIR=<llama.cpp at 3cf03257>
  -DCMAKE_C_COMPILER=/opt/rocm/llvm/bin/clang -DCMAKE_CXX_COMPILER=/opt/rocm/llvm/bin/clang++` then
  `ninja -C build-906 strata`: 158 of 158 steps, no patches.
- Load and serve: `strata --serve ... --max-context 65536` answers `/health` with `max_context: 65536`; the standard
  run started from that fresh engine and reported no earlier requests.
- `strata bench run --suite standard-v1`: status complete, 21/21 requests, 0 failed, 0 skipped, 12/12 sanity checks.
- The ten quick-suite runs: 0 failed requests.

**Failed:**
- none observed. The standard run excludes one request from its speed summaries because a first-answer timestamp was not
  available (`performance-prompt-4k-r3`); the report records it.

**Not run:**
- pictures (`--vision`: no `mmproj` on this card), `--clients`, `--long`, a second build for comparison, a long soak.
- A standard-protocol run at 262144 context: the protocol's cases are 4K and 16K, and this card's 262144 context streams
  the KV, so only 65536 is covered by the protocol run; the 262144 numbers in the sweep are quick-suite numbers.
- No conformance claim beyond the runner's own `complete` status.

## Limits

- One card, one CPU, one machine; nothing here is a claim about other gfx906 cards.
- The protocol is a draft, and its track is `custom`, so this is not a headline "standard" number.
- The quick-suite sweep uses `temperature: 0` with thinking off and no sanity checks; it isolates one setting and is not
  comparable with the protocol's numbers.
- The two contexts in the sweep are not comparable with each other (65536 keeps the KV in VRAM, 262144 streams it).
- The agent test in the sweep carries more run-to-run noise than its chat test.

Related on this site

Quick links to related pages.