#6295 [Benchmark] v0.2.16 on 2x Arc Pro B60 (layer split) and 1x Arc Pro B70 / Qwen3.8-Flash-Next UD-Q4_K_XL / 262K: same speed as v0.2.13, `!!!!` with --pipeline-windows 2 back to 0 of 12
open · @mbaaku · 0 comentários · No GitHub
BenchmarksSetup & installServer & APIMulti-GPUModels & quantsDocumentationWindows
Descrição
2x Arc Pro B60 (layer split) and 1x Arc Pro B70 / Qwen3.8-Flash-Next UD-Q4_K_XL / 262K / v0.2.16: standard runs on both hosts, context sweep to 262K on the B70, and `!!!!` output with `--pipeline-windows 2` on a layer split (follow-up to #5583)
**Summary**
- Both hosts moved to v0.2.16 (`64524fb9`, SYCL engine 0.1.62, source build), with the same model files (UD-Q4_K_XL, identical SHA-256), context (262,144) and engine options as in #5583. The standard benchmark (`strata bench run`) finished with **Status: complete** and **12 of 12** sanity checks on both hosts.
- **Speed is unchanged against v0.2.13 (engine 0.1.59), within run-to-run noise.** The move to the engine's own code (no ggml in the release engine) did not change speed on either host.
- 2x B60, layer split, MTP, `--pipeline-windows 0`: prompt 615 / 1,004 tok/s (4K / 16K), decode 30-32 tok/s. v0.2.13: 612 / 1,004, decode 30-31.
- 1x B70, DFlash2: prompt 1,361 / 1,717 tok/s, decode 40-48 tok/s. v0.2.13: 1,362 / 1,716, decode 40-48.
- Context sweep on the B70 to 262K tokens: prompt read 1,421-1,556 tok/s from 36K up, decode 36-42 tok/s. Same as v0.2.13 (1,406-1,544, 34-40).
- **2x B60 with the default `--pipeline-windows 2`: the `!`-only output is back to the v0.2.12 level.**
- v0.2.16: **0 of 12** sanity checks; all 12 sanity answers have reasoning made of `!` up to the 2,048-token cap (11 of them nothing but `!`), and one 16K answer has a run of 75 `!` in its final answer.
- v0.2.13, same test: 8 of 12 (4 of 22 answers). v0.2.12: 0 of 12 (20 of 22).
- `--pipeline-windows 0` still avoids it: 12 of 12 on v0.2.16.
- **DFlash2 on a layer split (section 5 of #5583) is fixed:** Strata now picks the MTP directly ("Drafting: MTP (DFlash2 isn't available in this mode)"), with no failed engine start and no exit 139.
- Getting v0.2.16 to run on Intel needed three local fixes for the new `engine/` / `terminal/` layout. Two of them hit an unmodified clone: the SYCL setup cannot start the server, and `strata bench run` from a git clone is always "nonconforming". See the separate issue ([link](https://github.com/Niko1221/Strata/issues/6296)).
---
#### Hardware and software
Same machines, model files, drivers and engine containers as in #5583. Changes:
| | Host A (2x B60) | Host B (1x B70) |
|---|---|---|
| Strata | v0.2.16 `64524fb9`, engine 0.1.62, source build (AOT `bmg-g21`) | v0.2.16 `64524fb9`, engine 0.1.62, source build (AOT `bmg-g31`) |
| Kernel | 7.0.0-43-generic | 7.1.8+deb13-amd64 |
| Docker / Compose | 29.9.3 / 5.7.0 | 29.9.3 / 2.26.1 |
| Engine container | same image as in #5583 | same image as in #5583 |
| Placement | card 0 expert cache 5,975 slots (17.39 GiB), 12,457 experts mirrored in pinned host memory (36.28 GiB); card 1 6,144 of 6,144 slots (18.07 GiB); MTP 865 MiB; 1,956 MiB VRAM free | expert cache 7,741 slots (22.56 GiB), 51.42 GiB mirrored; DFlash2 drafter 869 MiB; 1,743 MiB VRAM free |
| PCIe probe | 14.2 GB/s host to device | 23.6 GB/s host to device |
| ZFS (Host B) | - | dataset `recordsize=16K`, `primarycache=all` (section 3 of #5583) |
<details>
<summary>Engine command lines (from the engine log) and local patches</summary>
**Host A (2x B60):**
```
--serve --expert-cache auto --spec 2 --spec-min-p 0.5 --mtp /work/data/mtp/rt --max-context 262144 --kv-resident 32768
--pack ... --native ...-00001-of-00004.gguf --expert-profile .../terminal/data/expert-profile-sycl.bin --stream-experts
--vram-reserve-mib 2048 --prefill 8192 --kv fp16 --pipeline-windows 0 --layer-split 36
```
Section 4 changes only `--pipeline-windows 0` to `2`.
**Host B (1x B70):**
```
--serve --expert-cache auto --max-context 262144 --kv-resident 32768 --pack ... --native ...-00001-of-00004.gguf
--expert-profile .../terminal/data/expert-profile-sycl.bin --stream-experts --prefill 8192 --kv fp16 --spec 4
--vram-reserve-mib 1536 --draft dflash2 --draft-model /work/data/dflash2/qwen3.8-flash-next-dflash2.q4_0.pack
--draft-fallback none --mtp /work/data/mtp/rt --draft-conf 0.6
```
Environment as in #5583 (`STRATA_PLE_IO=ssd` on both).
**Local patches** (`patches/v0.2.16/`, applied with `git apply` in this order):
- `01-sycl-path-idempotent`, `02-param-forms` (narrowed: the double-quoted and unquoted forms are now upstream's, #5969), `03-invoke-forms` (reworked on top of #5969), `04-ple-io-env`, `05-dump-prompts`, `06-user-args-dedupe`
- new for the 0.2.16 layout: `07-sycl-repo-in-root`, `08-sycl-start-script-path`, `09-bench-commit-layout` (see the separate issue)
- dropped, fixed upstream: the two drafting path patches from #5583 (`drafting._sycl_host_path`, #5395; the engine binary behind the Intel launcher)
</details>
---
#### 1. Standard benchmark (`strata bench run`, protocol `strata-qwen38-thinking-v1-draft.1`)
Both runs ran inside the engine container against a freshly loaded engine. Both have **Status: complete** and **12 of 12** sanity checks.
```sh
strata bench run --url http://127.0.0.1:8080 --track custom --quantization UD-Q4_K_XL --track-notes "..."
```
Sampling follows the protocol (thinking on, temperature 1.0, top_p 0.95, top_k 20, reasoning_effort xhigh). Values are median (min-max), n=3.
**Host A, 2x B60, layer split, MTP, `--pipeline-windows 0`** (run `20261119T124823Z-61ce296e`)
| Case | Prompt tok/s | Decode tok/s | TTFT s | First answer s | Total s | Capped |
| --- | --- | --- | --- | --- | --- | --- |
| short | n/a | 30.4 (30.0-31.5) | 1.91 (1.77-1.95) | 6.17 (4.62-7.73, n=2) | 14.3 (12.9-18.8) | 1/3 |
| prompt-4k | 614.6 (612.3-614.8) | 31.6 (29.8-31.6) | 6.71 (6.71-6.74) | 17.15 (13.75-20.75) | 22.9 (20.8-23.9) | 2/3 |
| prompt-16k | 1003.8 (1003.2-1004.8) | 30.3 (29.6-30.8) | 16.38 (16.36-16.38) | 30.18 (n=1) | 33.2 (32.9-33.6) | 3/3 |
An earlier run in the same session (`20261119T123958Z-b28dfd26`, nonconforming only because of the commit bug in the separate issue) gave prompt 616.5 / 1007.1 tok/s, decode 31.7 / 30.4 / 30.3 tok/s, 12 of 12.
**Host B, 1x B70, DFlash2** (run `20261119T134945Z-f516fef8`)
| Case | Prompt tok/s | Decode tok/s | TTFT s | First answer s | Total s | Capped |
| --- | --- | --- | --- | --- | --- | --- |
| short | n/a | 48.3 (46.2-49.4) | 0.71 (0.71-0.71) | 2.62 (2.23-2.64) | 8.3 (6.4-10.2) | 0/3 |
| prompt-4k | 1360.7 (1333.2-1364.5) | 40.4 (39.9-45.1) | 3.04 (3.03-3.11) | 10.54 (6.99-14.09, n=2) | 15.7 (13.2-15.8) | 2/3 |
| prompt-16k | 1716.7 (1546.7-1717.0) | 42.6 (38.9-44.5) | 9.59 (9.58-10.63) | 18.05 (15.29-21.27) | 22.6 (21.1-22.7) | 3/3 |
**Against v0.2.13** (same hosts, files and options; runs `20261109T165822Z-47f33fac` and `20261109T165512Z-e5302597`)
| | Host A v0.2.13 | Host A v0.2.16 | Host B v0.2.13 | Host B v0.2.16 |
|---|---:|---:|---:|---:|
| prompt-4k, prompt tok/s | 612.1 | 614.6 | 1362.1 | 1360.7 |
| prompt-16k, prompt tok/s | 1003.7 | 1003.8 | 1716.3 | 1716.7 |
| decode short / 4k / 16k, tok/s | 31.2 / 29.7 / 30.7 | 30.4 / 31.6 / 30.3 | 47.8 / 40.5 / 42.8 | 48.3 / 40.4 / 42.6 |
| TTFT 16k, s | 16.38 | 16.38 | 9.59 | 9.59 |
| sanity checks | 12 / 12 | 12 / 12 | 12 / 12 | 12 / 12 |
"Capped" means the request reached the protocol's 512-token cap, counting reasoning and answer together; with reasoning_effort xhigh this is expected for the 4K/16K cases. The `results.json`, `outputs.jsonl` and README folders are available on request.
---
#### 2. Context sweep (style of #5138), Host B only
Same method and script as #5583 section 2: the repo's `docs/*.md` repeated to length with a unique first line (the engine reported `0 reused` for every request), greedy, 300 tokens streamed, two runs per length. Decode is measured by the client and agrees with the engine's figure within 0.1 tok/s. Host A was not swept this time (its GPUs serve another model); its standard run matches v0.2.13.
**Host B, 1x B70, DFlash2 `--spec 4`, v0.2.16**
| Prompt tokens (run 1 / 2) | Prompt read, tok/s | Time to first token, s | Decode, tok/s | Drafts accepted |
|---:|---:|---:|---:|---:|
| 1,012 / 953 | 626 / 605 | 1.6 / 1.6 | 42.5 / 43.1 | 84% / 84% (320/381) |
| 8,287 / 8,505 | 1,405 / 1,298 | 5.9 / 6.6 | 38.0 / 38.0 | 84% / 83% (249/297) |
| 36,388 / 36,591 | 1,511 / 1,556 | 24.1 / 23.6 | 41.5 / 40.6 | 85% / 83% (301/358) |
| 68,370 / 68,199 | 1,535 / 1,545 | 44.6 / 44.2 | 37.0 / 37.7 | 78% / 76% (296/385) |
| 128,719 / 128,993 | 1,484 / 1,500 | 86.9 / 86.1 | 37.0 / 37.3 | 79% / 79% (308/388) |
| 206,776 / 207,653 | 1,446 / 1,435 | 143.2 / 144.9 | 35.8 / 38.1 | 80% / 83% (364/447) |
| - / 261,801 | - / 1,422 | - / 184.3 | - / 38.8 | - / 88% (183/208) |
The first 250K-target request was rejected with HTTP 400 before reading; most likely its prompt plus the 300 output tokens did not fit the 262,144-token context (the second one, with a different starting point, was 261,801 tokens).
Same sweep on v0.2.13 (2026-11-09), run 1 / 2: prompt read 626/609, 1,421/1,300, 1,518/1,544, 1,455/1,544, 1,450/1,516, 1,426/1,444, 1,406/1,413 tok/s; decode 48.4/42.5, 39.9/35.5, 39.9/39.4, 37.5/38.2, 37.1/36.3, 36.7/34.3, 37.4/36.0 tok/s. Nothing moved beyond noise.
---
#### 3. `!!!!` output on a layer split with `--pipeline-windows 2` (2x B60)
Same setup as Host A above; the only change is the default `--pipeline-windows 2`. Fresh engine (run `20261119T135835Z-c55d4f00`).
| Version | `--pipeline-windows` | Answers with reasoning of only `!` | Other runs of `!` (≥ 20) | Sanity checks | Prompt 4K / 16K, tok/s | Decode short / 4K / 16K, tok/s |
|---|---:|---:|---|---:|---:|---:|
| v0.2.12 (#5583) | 2 | 20 of 22 | - | **0 of 12** | 983.9 / 1450.0 | 32.7 / 31.8 / 31.3 * |
| v0.2.13 | 2 | 4 of 22 (I1-I4) | one 16K answer | 8 of 12 | 616.0 / 1003.7 | 38.1 / 33.2 / 35.4 |
| **v0.2.16** | 2 | **11 of 22** (I2-T4) | I1 (normal start, then `!` to the cap); one 16K answer with 75 `!` in the final answer | **0 of 12** | 616.8 / 1008.2 | 36.9 / 33.9 / 33.9 |
| v0.2.16 | 0 | 0 of 22 | none | 12 of 12 | 614.6 / 1003.8 | 30.4 / 31.6 / 30.3 |
\* v0.2.12: mostly degenerate output.
- **What it looks like on v0.2.16:** the warm-up and the nine performance requests are normal (one 16K answer has 75 `!` in the middle of its final answer). From the first sanity check (I1) on, every answer's reasoning turns into `!!!!` up to the 2,048-token cap, with an empty final answer and no tool call. The engine reports no error; every one is "truncated".
- **Decode on pure `!`** rises to 55-58 tok/s (the drafts are always accepted), as on v0.2.12.
- On the normal answers, `--pipeline-windows 2` decodes about 15% faster than `0` (34-37 against 30-32 tok/s), as on v0.2.13; prompt reading is the same (the 984 / 1,450 tok/s of v0.2.12 have not come back since v0.2.13).
- So v0.2.16 is worse than v0.2.13 here and back at the v0.2.12 level. Single card (Host B) is not affected. We can run a debug build or extra logging on this host if that helps (`STRATA_DEBUG=1`, `STRATA_DBG_NAN=1`, ...).
---
#### 4. DFlash2 on a layer split: fixed
On Host A, v0.2.16 logs `[strata] Drafting: MTP (DFlash2 isn't available in this mode)` and starts the engine once with `--spec 2 --mtp ...`. The SYCL engine still refuses `--draft dflash2` with more than one card (`engine/sycl/src/program/generate.cpp`: "is for one card and one request at a time (not --batch, --pipeline-windows or a layer split)"), but the plan no longer tries it, so the exit 139 and the extra engine start from #5583 section 5 are gone.
No site
Links install, modelos, releases.