#6413 [Benchmark] v0.2.17 on 2x Arc Pro B60 (layer split) and 1x Arc Pro B70 / Qwen3.8-Flash-Next UD-Q4_K_XL / 262K: same speed as v0.2.16, `--pipeline-windows 2` no longer produces `!!!!` (12 of 12)

open · @mbaaku · 0 comentarios · En GitHub

BenchmarksSetup & installServer & APIMulti-GPUAMD / HIPModels & quantsDocumentationWindows

Descripción

x Arc Pro B60 (layer split) and 1x Arc Pro B70 / Qwen3.8-Flash-Next UD-Q4_K_XL / 262K / v0.2.17: standard runs on both hosts, and the layer split with the default `--pipeline-windows 2` (follow-up to #6295 and #5583)

**Summary**

- Both hosts moved to v0.2.17 (`12d491da`, SYCL engine 0.1.63, source build), with the same model files (UD-Q4_K_XL, identical SHA-256), context (262,144) and engine options as in #6295. The standard benchmark (`strata bench run`) finished with **Status: complete** on both hosts.
- **Speed is unchanged against v0.2.16 (engine 0.1.62), within run-to-run noise.**
  - 2x B60, layer split, MTP, `--pipeline-windows 0`: prompt 618 / 1,003 tok/s (4K / 16K), decode 30-31 tok/s. v0.2.16: 615 / 1,004, decode 30-32.
  - 1x B70, DFlash2: prompt 1,360 / 1,714 tok/s, decode 40-48 tok/s. v0.2.16: 1,361 / 1,717, decode 40-48.
- **2x B60 with the default `--pipeline-windows 2`: the `!`-only output is gone.**
  - v0.2.17: **12 of 12** sanity checks, and not a single `!` in any of the 22 answers.
  - v0.2.16, same test: 0 of 12 (11 of 22 answers with reasoning of only `!`). v0.2.13: 8 of 12. v0.2.12: 0 of 12.
  - Decode with `--pipeline-windows 2` is 7-17% faster than with `0` (35.4 / 32.1 / 33.8 against 30.3 / 30.1 / 30.7 tok/s); prompt reading is the same.
  - This is one run of 22 answers, not a soak. We will run it in real use for longer and report back if `!` shows up again.
- Expert placement is the same as on v0.2.16 on both hosts.

---

#### Hardware and software

| | Host A (2x B60) | Host B (1x B70) |
|---|---|---|
| GPU | 2x Intel Arc Pro B60 24 GB | 1x Intel Arc Pro B70 32 GB (PCIe 5.0 x16) |
| CPU / RAM | AMD Ryzen 5 7600X, 128 GB DDR5 (123 GB usable) | AMD Ryzen 9 7950X3D, 96 GB DDR5-6000 |
| OS / kernel | Ubuntu 26.04 LTS, 7.0.0-43-generic | Debian 13, 7.1.8+deb13-amd64 |
| Docker / Compose | 29.9.3 / 5.7.0 | 29.9.3 / 2.26.1 |
| Strata | v0.2.17 `12d491da`, engine 0.1.63, source build (AOT `bmg-g21`) | v0.2.17 `12d491da`, engine 0.1.63, source build (AOT `bmg-g31`) |
| Engine container | same image as in #5583 / #6295 | same image as in #5583 / #6295 |
| Model | UD-Q4_K_XL, 4 files, 262,144 context, KV fp16 | same files (identical SHA-256) |
| Drafting | MTP `--spec 2` (DFlash2 is not available on a layer split) | DFlash2 Q4_0, `--spec 4`, `--draft-conf 0.6` |
| Placement | card 0: layers 0-35, expert cache 5,975 slots (17.39 GiB), 12,457 experts mirrored in pinned host memory (36.28 GiB); card 1: layers 36-47, 6,144 of 6,144 slots (18.07 GiB); MTP 865 MiB; 1,956 MiB VRAM free. With `--pipeline-windows 2`: card 0 5,865 slots (17.07 GiB) | expert cache 7,741 slots (22.56 GiB), 51.42 GiB mirrored; DFlash2 drafter 869 MiB; 1,747 MiB VRAM free |
| PCIe probe | 14.2 GB/s host to device | 23.4 GB/s host to device |
| Storage | NVMe | NVMe, ZFS dataset `recordsize=16K`, `primarycache=all` (section 3 of #5583) |

<details>
<summary>Engine command lines (from the engine log)</summary>

**Host A (2x B60):**

```
--serve --expert-cache auto --spec 2 --spec-min-p 0.5 --max-context 262144 --kv-resident 32768
--pack ... --native ...-00001-of-00004.gguf --expert-profile .../terminal/data/expert-profile-sycl.bin --stream-experts
--vram-reserve-mib 2048 --prefill 8192 --kv fp16 --pipeline-windows 0 --mtp /work/data/mtp/rt --layer-split 36
```

Section 2 changes only `--pipeline-windows 0` to `2`.

**Host B (1x B70):**

```
--serve --expert-cache auto --max-context 262144 --kv-resident 32768 --pack ... --native ...-00001-of-00004.gguf
--expert-profile .../terminal/data/expert-profile-sycl.bin --stream-experts --prefill 8192 --kv fp16 --spec 4
--vram-reserve-mib 1536 --draft dflash2 --draft-model /work/data/dflash2/qwen3.8-flash-next-dflash2.q4_0.pack
--draft-fallback none --mtp /work/data/mtp/rt --draft-conf 0.6
```

</details>

---

#### 1. Standard benchmark (`strata bench run`, protocol `strata-qwen38-thinking-v1-draft.1`)

Each run ran inside the engine container against a freshly loaded engine.

```sh
strata bench run --url http://127.0.0.1:8080 --track custom --quantization UD-Q4_K_XL --track-notes "..."
```

Sampling follows the protocol (thinking on, temperature 1.0, top_p 0.95, top_k 20, reasoning_effort xhigh). Values are median (min-max), n=3.

**Host A, 2x B60, layer split, MTP, `--pipeline-windows 0`** (run `20261120T153844Z-2f5d945c`), **Status: complete**, sanity checks 11 of 12

| Case | Prompt tok/s | Decode tok/s | TTFT s | First answer s | Total s | Capped |
| --- | --- | --- | --- | --- | --- | --- |
| short | n/a | 30.3 (30.3-31.1) | 1.92 (1.81-1.93) | 4.89 (4.85-6.61) | 11.8 (11.0-12.9) | 0/3 |
| prompt-4k | 617.9 (602.5-618.0) | 30.1 (29.4-30.7) | 6.67 (6.67-6.87) | 15.03 (n=1) | 23.6 (23.5-24.0) | 3/3 |
| prompt-16k | 1003.1 (999.7-1008.0) | 30.7 (29.2-31.6) | 16.40 (16.31-16.45) | 27.21 (21.61-32.82, n=2) | 33.0 (32.6-33.8) | 3/3 |

The one failed check is T2 (`get_weather`): the model wrote the call inside its reasoning, then imagined an empty tool result there and answered "the lookup returned no weather data" without making the call. Fed through the v0.2.17 output parser again, the same text gives no call either, so we read it as the model at temperature 1.0, not a parser problem. An earlier run on the same build had 12 of 12, with prompt 615.3 / 1001.4 tok/s and decode 30.7 / 29.7 / 30.2 tok/s.

**Host B, 1x B70, DFlash2** (run `20261120T153334Z-6aa6e1a1`), **Status: complete**, sanity checks **12 of 12**

| Case | Prompt tok/s | Decode tok/s | TTFT s | First answer s | Total s | Capped |
| --- | --- | --- | --- | --- | --- | --- |
| short | n/a | 48.1 (46.2-49.4) | 0.71 (0.71-0.71) | 2.61 (2.23-2.64) | 8.3 (6.4-10.2) | 0/3 |
| prompt-4k | 1359.7 (1331.3-1362.8) | 40.4 (40.0-45.0) | 3.04 (3.04-3.11) | 10.56 (7.01-14.11, n=2) | 15.7 (13.3-15.8) | 2/3 |
| prompt-16k | 1714.4 (1621.5-1715.5) | 42.6 (38.5-44.3) | 9.60 (9.59-10.15) | 17.56 (15.28-21.32) | 22.1 (21.1-22.9) | 3/3 |

An earlier run on the same build gave 1362.2 / 1717.0 tok/s, decode 48.1 / 40.4 / 42.6 tok/s, 12 of 12.

**Against v0.2.16** (#6295; same hosts, files and options)

| | Host A v0.2.16 | Host A v0.2.17 | Host B v0.2.16 | Host B v0.2.17 |
|---|---:|---:|---:|---:|
| prompt-4k, prompt tok/s | 614.6 | 617.9 | 1360.7 | 1359.7 |
| prompt-16k, prompt tok/s | 1003.8 | 1003.1 | 1716.7 | 1714.4 |
| decode short / 4k / 16k, tok/s | 30.4 / 31.6 / 30.3 | 30.3 / 30.1 / 30.7 | 48.3 / 40.4 / 42.6 | 48.1 / 40.4 / 42.6 |
| TTFT 16k, s | 16.38 | 16.40 | 9.59 | 9.60 |
| sanity checks | 12 / 12 | 11 / 12 (T2, see above) | 12 / 12 | 12 / 12 |

"Capped" means the request reached the protocol's 512-token cap, counting reasoning and answer together; with reasoning_effort xhigh this is expected for the 4K/16K cases. The `results.json`, `outputs.jsonl` and README folders are available on request.

---

#### 2. `--pipeline-windows 2` on a layer split (2x B60): no more `!!!!`

Same setup as Host A above; the only change is the default `--pipeline-windows 2`. Fresh engine (run `20261120T153043Z-45ff023a`), **Status: complete**, sanity checks **12 of 12**.

| Version | `--pipeline-windows` | Answers with reasoning of only `!` | Other runs of `!` (≥ 20) | Sanity checks | Prompt 4K / 16K, tok/s | Decode short / 4K / 16K, tok/s |
|---|---:|---:|---|---:|---:|---:|
| v0.2.12 (#5583) | 2 | 20 of 22 | - | 0 of 12 | 983.9 / 1450.0 | 32.7 / 31.8 / 31.3 * |
| v0.2.13 | 2 | 4 of 22 | one 16K answer | 8 of 12 | 616.0 / 1003.7 | 38.1 / 33.2 / 35.4 |
| v0.2.16 (#6295) | 2 | 11 of 22 | I1; one 16K answer | 0 of 12 | 616.8 / 1008.2 | 36.9 / 33.9 / 33.9 |
| **v0.2.17** | 2 | **0 of 22** | **none (no `!` in any answer)** | **12 of 12** | 617.3 / 1007.9 | 35.4 / 32.1 / 33.8 |
| v0.2.17 | 0 | 0 of 22 | none | 11 of 12 (T2) | 617.9 / 1003.1 | 30.3 / 30.1 / 30.7 |

\* v0.2.12: mostly degenerate output.

| Case (v0.2.17, `--pipeline-windows 2`) | Prompt tok/s | Decode tok/s | TTFT s | First answer s | Total s | Capped |
| --- | --- | --- | --- | --- | --- | --- |
| short | n/a | 35.4 (33.9-36.7) | 1.90 (1.88-1.99) | 5.05 (4.26-5.12) | 10.7 (10.4-11.7) | 0/3 |
| prompt-4k | 617.3 (594.4-618.8) | 32.1 (30.6-37.3) | 6.67 (6.66-6.93) | 15.49 (n=1) | 22.6 (20.4-23.6) | 3/3 |
| prompt-16k | 1007.9 (996.2-1009.5) | 33.8 (32.3-37.5) | 16.31 (16.28-16.50) | 27.81 (23.17-31.04) | 31.6 (29.9-32.1) | 3/3 |

- All 22 answers (warm-up, nine performance requests, twelve sanity checks) are normal text; the four tool-call checks returned the expected calls.
- Card 0 holds 110 fewer expert slots than with `--pipeline-windows 0` (5,865 against 5,975); the decode gain is the same as on v0.2.13 / v0.2.16 (about 10-15% on normal answers), and prompt reading does not change.
- The v0.2.17 release notes do not name this. Was it an intended fix (which change), or might it come back? One run of 22 answers is not a soak; we will keep testing `--pipeline-windows 2` on this host in real use and report back. Single card (Host B) was not affected in any version.

En el sitio

Enlaces a install, modelos, releases.