#6521 [Benchmark] R9700 + Threadripper PRO 3975WX: Hy4-Flash at ~30 tok/s (SQ-64, 4K) vs. 62 tok/s in the 0.2.18 notes

open · @ohrensessel · 3 Kommentare · Auf GitHub

BenchmarksSetup & installServer & APIAMD / HIPModels & quantsLinux

Beschreibung

The 0.2.18 release notes list 62 tok/s for Hy4-Flash on R9700 / RX 9070 XT. On my R9700 with Hy4-Flash SQ-96 I measure about 23 tok/s decode.

**Hardware / OS**
- GPU: AMD Radeon AI PRO R9700, 32 GB; amdgpu 31.70
- CPU: AMD Ryzen Threadripper PRO 3975WX, one NUMA node
- RAM: 124 GiB DDR4; model files on a RAID 0 of three PCIe 3.0 NVMe SSDs
- OS: Ubuntu 26.04.1, kernel 7.0.0-44

**Strata version / model**
- Strata 0.2.18 (built from the `v0.2.18` tag), engine component `strata-hy4-0.1.64-linux-x86_64-hip-rx9070` from the release
- Hy4-Flash SQ-96 with Engram-Q8 and Vision; defaults otherwise; no other model on the GPU, machine otherwise idle

**Commands**
- Setup: `.venv/bin/python setup.py --setup --yes --backend hip --family hy4-flash --model SQ-96 --gguf-dir <folder with the three files> --context 131072 --host 0.0.0.0 --port 8099 --no-start`
- Start: `.venv/bin/python setup.py --port 8099`
- Measurement: `/v1/chat/completions`, non-streaming, one request at a time, 8 short prompts, `max_tokens` 512, `chat_template_kwargs: {"enable_thinking": false}`; tok/s = `usage.completion_tokens` / wall time, median of 8

**Numbers**

| test | result |
|---|---|
| decode 8 x 512, `temperature` 0 | 22.7 / 22.9 tok/s (2 runs) |
| decode 8 x 512, `temperature` 0.9, `top_p` 1.0 | 22.3 / 21.9 tok/s (2 runs) |
| prompt reading, 120K tokens | 1,046 tok/s |

No failed requests, no errors in the engine log.

**Engine log (excerpts)**

```
[args] --vram-experts-gb 17.11: not used by this runner yet
[args] --ram-experts-gb 65.82: not used by this runner yet
[load] experts tiered: 0 static VRAM (0.00 GB) + 4590 adaptive VRAM slots (22.45 GB) = 22.45 GB VRAM, 20992 in host RAM computed by the CPU (82.93 GB), 0.1 s
[cpuexp] THP: AnonHugePages 77.23 GB -> 77.23 GB of 77.24 GB arena (collapse off: 0 MiB ok, 0 MiB failed, 0.0 s)
[cpuexp] 20992 experts resident, 82.93 GB in 14.2 s (5.84 GB/s), 626 reads, mlock=yes, isa=scalar, kernel=fast(avx2), act=hi(2xQ8_1)
[cpuexp] 32 pool threads (pinned, physical cores first), spin 300 us, sched=1, pf=2 rows
[spec] verify steps 881: accepted drafts per step 2.493 (verified slots 4.89/step, ...), draft 4.0 ms/step, verify 146.1 ms/step, plain 120.0 ms/st
```

### Reproduction status
Measured locally

Mehr auf der Site

Links zu Install, Modellen, Releases.