#2859 [Community benchmark] RX 9070 16 GB + Ryzen 7 5700 + 32 GiB RAM: Q2_0 at ~65 tok/s on Windows HIP 0.1.44.1

Description

## Summary

Sharing measurements from my tuned Q2_0 setup for others with similar hardware, especially a 32 GiB RAM system.

Short-chat decode reached a median of 64.7 tok/s. Three longer generations, each producing 1,024 tokens, measured 64.2 / 64.9 / 69.1 tok/s. Fresh 16K and 32K prompts took about 73 and 105 seconds to the first token, respectively. Follow-up conversations benefited from prefix reuse.

Measured on 2026-10-17. These results describe the combined configuration below.

## Hardware and software

- GPU: AMD Radeon RX 9070, 16 GB, gfx1201; engine reports 16,304 MiB.
- CPU: AMD Ryzen 7 5700, 8 cores / 16 threads, AVX2.
- RAM: 4 x 8 GiB DDR4, configured at 2666 MT/s; 31.80 GiB OS-visible.
- Model storage: KINGSTON SA2000M8500G, 500 GB NVMe SSD.
- OS: Windows 11 Home, build 26300.
- AMD driver: 32.0.32015.2008.
- Strata engine: 0.1.44.1, Windows x64 HIP prebuilt.
- Bundled ROCm: 10.2.0a20260930; hipBLASLt: 100500.
- Python: project .venv, 3.14.6.
- Normal desktop background workload; no power-setting changes during the test.

## Model and settings

Original Flash-Next GSQ-RCO Q2_0, with these source files:

- Qwen3.8-Flash-Next-GSQ-RCO-Q2_0-00001-of-00002.gguf
- Qwen3.8-Flash-Next-GSQ-RCO-Q2_0-00002-of-00002.gguf

Engine settings:

```text
--expert-cache auto
--prefill auto
--spec 3
--spec-min-p 0.5
--max-context 65536
--kv q4_0
--mmap-experts
--pcie-frac 0.20
--vram-reserve-mib 1500
--host-core last
--adapt-every 8
--adapt-swaps 48
--control-vector-scaled data/experimental-speed-projection/Qwen3.8-Flash-Next-experimental-speed-projection.gguf:1.0
--control-vector-layer-range 4 44
--cvec-mode project
--cvec-dir per-layer
```

Expert profile: the repository's data/expert-profile.bin.
MTP: local Q2_0-expert runtime in Strata-data/mtp/rt, with mixed Q8_0/BF16/F32 dense tensors. MTP max is 3; lookup drafts can extend the verify window to 5.

Environment overrides:

```json
{
  "STRATA_HIPBLASLT_TUNING": "tools/hip/gfx1201-hipblaslt-100500.txt",
  "STRATA_FILE_RELEASE": "1",
  "GPU_PINNED_MIN_XFER_SIZE": "1048576",
  "STRATA_PREFILL_CPU_SHARE": "auto",
  "STRATA_PF_FUSED": "0",
  "STRATA_STAGER_THREADS": "4",
  "STRATA_ADAPT_LAG": "2",
  "STRATA_ROUTE_PRIOR": "0.5"
}
```

Actual startup allocation:

- Expert cache: 6,273 slots / 8,270 MiB.
- Prefill chunk: 6,144 tokens.
- Prompt path borrows 1,957 cache slots / 2.52 GiB.
- Free VRAM after loading: 1,191 MiB.
- CPU pool: 7 workers plus the host thread, with --host-core last.
- Active RDNA4 defaults logged: STRATA_HC_Q8=1 and STRATA_ROUTE_TAIL_SKIP=7.

STRATA_ROUTE_PRIOR=0.5 biases expert selection toward resident experts, and the projection changes residual activations; both are part of the measured setup.

## Results

Median (min–max). Prompt/decode speeds are from API engine timings; time to first token (TTFT) is measured by the client.

| Workload | Requests | Actual prompt tokens | Output tokens | Prompt tok/s | Decode tok/s | TTFT seconds |
|---|---:|---:|---:|---:|---:|---:|
| Short chat | 9 | 43–51 | 100 | 105.5 (45.8–118.7) | 64.7 (36.8–72.2) | 0.480 (0.425–0.960) |
| Fresh 4K prompt | 3 | 4,019 | 24 | 187.8 (143.2–193.2) | 42.3 (32.8–45.9) | 21.434 (20.843–28.093) |
| Fresh 16K prompt | 3 | 16,058 | 24 | 219.8 (217.4–245.1) | 39.6 (37.0–43.1) | 73.106 (65.580–73.901) |
| Fresh 32K prompt | 3 | 32,049 | 24 | 304.6 (290.1–305.6) | 46.5 (41.9–57.7) | 105.273 (104.915–110.545) |
| Agent follow-up | 15 | 3,741–14,557 | 5–64 | 226.4 (184.0–429.3) | 63.1 (41.4–100.6) | 12.791 (4.645–17.266) |
| Longer generation | 3 | 97 | 1,024 | 72.1 (47.9–85.5) | 64.9 (64.2–69.1) | 1.368 (1.156–2.050) |

Short-chat decode medians by round: 37.6 / 66.6 / 64.7 tok/s. All three rounds are included in the overall result.

Agent prefix reuse was 68.72% (68.69–68.74%) per session. Only 2–4 old-conversation tokens were re-read per session. On the final follow-up turn, total context was about 14.5K tokens, with 2,899 fresh tokens; median TTFT was 6.825 seconds.

## Method and checks

- Ran tools/strata_bench.py with --runs 3 --long.
- One server session, one GPU, sequential requests, temperature 0, thinking off.
- Four warm-up/calibration requests excluded from the measured results; model loading also excluded.
- Fresh prompts had unique tags and zero reused tokens. Expert-cache state and OS file-cache state were retained within the session.
- Agent workload: three sessions, each with one initial turn and five document-addition turns; tool schemas were supplied, with instructions not to call tools.
- Additional long-output test: three requests for a detailed local-inference performance guide, each capped at and producing 1,024 tokens.
- Official suite: 0 failed requests, 0 skipped tests.
- Arithmetic, Chinese arithmetic and JSON-format checks: 3/3 passed.
- Official needle_bench.py, --lengths 4k --depths 10,50,90: 3/3 found; actual prompt lengths were 3,327 / 3,327 / 3,328 tokens.

Whole-system telemetry sampled approximately every five seconds showed RAM usage up to 31.56 GiB. The engine startup log reported 23.6 GiB of mapped expert pages for CPU prefill sharing versus 22.7 GiB of free RAM. The configuration was kept unchanged throughout the test.

The practical result on this machine is roughly 65 tok/s for generation, with longer initial waits for fresh documents and shorter waits when continuing a cached conversation. The allocated context was 64K; fresh-prompt measurements reached about 32K.