#6614 [Engine] Hy4-Flash SQ-64 on RX 9070 XT 16 GB: 18-24 tok/s decode against 62 tok/s on the model page

open · @Vyeche · 0 コメント · GitHub で見る

BenchmarksSetup & installServer & APIAMD / HIPNVIDIA / CUDAModels & quantsLinux

本文

### Area

[Engine]  (speed, kernels, models, the server and API)

### Desktop app, Terminal or source?

Terminal (the `strata` command)

### What happened

Hy4-Flash SQ-64 on an RX 9070 XT 16 GB decodes at 18 to 24 tok/s here. The model page lists 62 tok/s decode and 1,122 tok/s prompt for "R9700 / RX 9070 XT" (4K context), and 44 tok/s for an RTX 5080 16 GB. I would like to know what setup the 9070 XT figure comes from, or whether something on this PC keeps it at a third of that.

Measured (temperature 0, one request each, `strata launch --family hy4-flash --model sq-64`, nothing else loaded, work VM off):

| Context | Request | Prompt | Decode |
|---|---|---:|---:|
| 128K | code, with thinking (400 tokens out) | 83 tok/s (40 tokens) | 21.4 tok/s |
| 128K | short story, `reasoning_effort: "none"` (447 out) | 79 tok/s | 19.2 tok/s |
| 32K | code, with thinking (674 out) | 79 tok/s (47 tokens) | 23.9 tok/s |
| 32K | short story (426 out) | 68 tok/s | 18.2 tok/s |
| 32K | question on a 7,827-token document (171 out) | 700 tok/s | 18.4 tok/s |
| 32K | tool-call request (96 out) | 71 tok/s (198 tokens) | 18.6 tok/s |

RAM in use with the model loaded: 67 of 91 GiB, no swap. `strata launch` offers 32K as the smallest context, so I did not test 4K.

What the engine log says about the placement (32K run):

```
[args] --vram-experts-gb 5.55: not used by this runner yet
[args] --ram-experts-gb 50.49: not used by this runner yet
[gpu] AMD Radeon RX 9070 XT gfx1201 free 15.8 / 15.9 GiB
[load] embed_tokens (0.38 GB) in pinned mapped host RAM -> trunk VRAM 5.54 GB
[load] experts tiered: 0 static VRAM (0.00 GB) + 1685 adaptive VRAM slots (6.25 GB) = 6.25 GB VRAM, 20992 in host RAM computed by the CPU (56.04 GB), 0.0 s
[cpuexp] 20992 experts resident, 56.04 GB in 8.9 s (6.27 GB/s), 421 reads, mlock=no, isa=scalar, kernel=fast(avx512-vbmi), act=q8
[cpuexp] 16 pool threads (pinned, physical cores first), spin 300 us, sched=1, pf=2 rows
[hybrid] 20992 experts (56.04 GB) in host RAM computed by 16 CPU threads
[spec] backoff state: backoffs 7, probes 7, plain-mode iterations 652, plain after draft (K'=0) 23, P now 256 (last 256)
[spec] verify ms by m: m2=190.0 m3=100.0 m4=219.7 m5=129.6 m6=133.2 m7=0.0 m8=176.2
```

So 6.25 GB of experts are on the card and 56 GB are computed by 16 CPU threads, the two placement arguments Control passes are not used, and DFly2 spent most steps in plain mode (652 plain iterations in this run; in the 128K run 0.9 to 1.4 accepted drafts per step, plain step about 53 to 56 ms).

One more observation from the same run, not investigated: the tool-call request (one `read_file` tool, "Open the file src/main.py and tell me what it does.") was answered with text ("I don't have access to your project's filesystem…") and no tool call. Qwen3.8-Flash-Next IQ3_S and Inkling-2-Mini SQ-10 call the tool for the same request.

Not tested: a 4K context, another thread count, the desktop moved off the card, SQ-96 (it does not fit 91 GiB of RAM: 90 GiB in use and 14 to 20 GB in swap after loading, 6.1 tok/s).

### Smallest steps or command that reproduce it

```sh
strata launch --family hy4-flash --model sq-64 --context 32k --vision on --tune skip --yes --no-tui --port 8081
curl -s http://127.0.0.1:8081/v1/chat/completions -H "Content-Type: application/json" -d '{"model":"x","messages":[{"role":"user","content":"Write a short story of about 300 words about a lighthouse keeper."}],"max_tokens":450,"reasoning_effort":"none","temperature":0}'
```

`timings.predicted_per_second` in the reply is 18 to 19.

### Reproduction status

Reproduced locally

### Strata version

Strata 0.2.18 (128K rows) and 0.2.19 (32K rows) · strata-hy4 0.1.64 (hip-rx9070 package)

### Graphics card, memory and system

Radeon RX 9070 XT 16 GB (gfx1201), 92 GB RAM, Ryzen 9 9950X3D (16 cores), Pop!_OS, Linux 7.2

### Model, size and settings

Hy4-Flash SQ-64 with Engram-Q8, 128K and 32K context, pictures on, DFly2 (built in), model files on NVMe

### Logs

See the engine log lines above.

This report was prepared with the help of an AI assistant; the measurements were run on the hardware above.

関連リンク

インストール・モデル・リリースへのサイト内リンク。