#5583 [Benchmark] 2x Arc Pro B60 (layer split) and 1x Arc Pro B70 / Qwen3.8-Flash-Next UD-Q4_K_XL / 262K / v0.2.12
closed · @mbaaku · 1 comentarios · En GitHub
BenchmarksSetup & installServer & APIMulti-GPUAMD / HIPModels & quantsDocumentationWindows
Descripción
2x Arc Pro B60 (layer split) and 1x Arc Pro B70 / Qwen3.8-Flash-Next UD-Q4_K_XL / 262K / v0.2.12: standard runs, context sweep to 259K, ZFS settings that cut long-prompt speed 3-4x, and `!!!!` output with `--pipeline-windows 2` on a layer split
**Summary**
- Two Intel hosts on v0.2.12 (`146df3c4`), the same model files (UD-Q4_K_XL, identical SHA-256) and the same 262,144-token context. The standard benchmark (`strata bench run`) finished with **Status: complete** and **12 of 12** sanity checks on both hosts.
- 2x Arc Pro B60, layer split, MTP: decode about 29-31 tok/s.
- 1x Arc Pro B70, DFlash2: decode about 37-45 tok/s.
- **2x B60, layer split: on v0.2.12 the model writes only `!` with the default `--pipeline-windows 2`.**
- In the standard run, 20 of the 22 answers have reasoning made of nothing but `!` up to the token cap, and 0 of 12 sanity checks pass.
- With `--pipeline-windows 0` it is 0 of 22 in two fresh runs, and 12 of 12 sanity checks pass both times.
- The cost is prompt reading: 30-37% slower at 4K/16K.
- We saw the same failure on v0.2.10 (engine 0.1.56) on this host, and the same workaround fixed it there.
- **Context sweep, prompts from about 1K to 259K tokens:**
- 2x B60: prompt read 1,057-1,116 tok/s from 32K up, decode 29-37 tok/s, with almost no drop up to 259K.
- B70: prompt read 1,425-1,554 tok/s from 36K up (259K in 181 s), decode 36-42 tok/s up to 259K.
- **With `STRATA_PLE_IO=ssd`, the filesystem under the model decides long-prompt speed.**
- B70 on a ZFS dataset with `recordsize=1M` and `primarycache=metadata`: 320-540 tok/s at 128K-259K, and the same length could vary 4x between runs.
- After `recordsize=16K` and `primarycache=all` (and rewriting the GGUF shard that holds the PLE table): 1,425-1,531 tok/s, stable. That is 3-4x faster at long context.
- See section 3.
- On a layer split, Auto still picks DFlash2. The engine refuses it ("for one card and one request at a time ... not ... a layer split") and exits with **139 (SIGSEGV)** instead of a clean error. Strata then restarts with the MTP, so it costs one extra engine start.
---
#### Hardware and software
| | Host A | Host B |
|---|---|---|
| GPU | 2x Intel Arc Pro B60 24 GB (`bmg-g21`, 8086:e211) | 1x Intel Arc Pro B70 32 GB (`bmg-g31`, 8086:e223) |
| GPU link | each card PCIe 4.0 x8 from the CPU (x8/x8; the card supports 5.0 x8); engine probe 14.2 GB/s host to device | PCIe 5.0 x16 from the CPU; engine probe 22.2 GB/s host to device |
| Display | integrated Radeon (Raphael) drives the desktop | integrated Radeon (Raphael) |
| CPU | AMD Ryzen 5 7600X, 6 cores / 12 threads, 32 MiB L3 | AMD Ryzen 9 7950X3D, 16 cores / 32 threads, 128 MiB L3 |
| RAM | 128 GB (122.7 GiB visible): 4x 32 GB DDR5 dual-rank, G.Skill F5-6400J3239G32G, **running at 3,600 MT/s** (kit rated 6,400 XMP, JEDEC 4,800; four dual-rank DIMMs on AM5) | 96 GB (93.4 GiB visible): 2x 48 GB DDR5 dual-rank, Corsair CMH96GX5M2B6000C30, 6,000 MT/s |
| Board / BIOS | ASUS ProArt B650-CREATOR / 4003 | ASUS ROG CROSSHAIR X670E HERO / 3902 |
| Model storage | Samsung SSD 990 PRO with Heatsink 4 TB (firmware 8B2QJXD7), CPU-attached M.2, ext4 (normal page cache, about 70 GiB of RAM free for it). The drive links at **PCIe 4.0 x1** (it supports x4) in this board/GPU lane configuration. | OpenZFS 2.4.4, pool of 2x2 mirrors of Samsung 870 EVO 4 TB SATA (ashift 13, no encryption, lz4), dedicated dataset for Strata; ARC max 12 GiB; dataset settings in section 3 |
| Host OS / kernel | Ubuntu 26.04.1 LTS, 7.0.0-42-generic, `xe` driver | Debian 13 (trixie), 7.1.8+deb13-amd64, `xe` driver |
| Docker | 29.9.2, containerd 2.4.2, Compose 5.6.1 | 29.8.1, containerd 2.3.5, Compose 2.26.1 |
| Engine container | Ubuntu 26.04 LTS, oneAPI DPC++ 2026.1.0 (20260617), compute-runtime 26.35.39758.10-0 (`intel-opencl-icd`, `libze-intel-gpu1`), level-zero 1.32.0; image built from `Dockerfile.b60`, AOT `bmg-g21` | same versions; AOT `bmg-g31` |
| Strata | v0.2.12 `146df3c4`, engine 0.1.58, source build | v0.2.12 `146df3c4`, engine 0.1.57, source build |
Both hosts carry the same 8 local patches on top of v0.2.12 (see the details block). They change the tool-call parser, the drafter path checks for a Docker-hosted server, and two environment switches. No other GPU work ran during the measurements.
<details>
<summary>Engine configuration (both hosts)</summary>
Model: Unsloth UD-Q4_K_XL, 4 GGUF files, SHA-256 `4448186216b3af4c…`, `3f342f1c1580473f…`, `56758f40269cad5c…`, `753bda48b98ba4f1…` (the same on both hosts). Vision off, one request at a time.
**Host A (2x B60):**
- Setup: `./setup.sh --backend sycl --family unsloth --model UD-Q4_K_XL --context 262144 --gpus 0,1 --layer-split 36 --kv fp16 --vision no`
- Engine: `--expert-cache auto --spec 2 --spec-min-p 0.5 --mtp data/mtp/rt --max-context 262144 --kv-resident 32768 --stream-experts --kv fp16 --vram-reserve-mib 2048 --prefill 8192 --pipeline-windows 0`
- `--pipeline-windows 0` is set in `user_args`.
- Without it the default is `--pipeline-windows 2`; see the failure section.
- Environment: `ONEAPI_DEVICE_SELECTOR=level_zero:*`, `STRATA_STAGE_TRIM=1`, `STRATA_IQ3S_WIDE=1`, `STRATA_VERIFY_DEVICE_PLAN=1`, `STRATA_VERIFY_NO_HOST=0`, `STRATA_PLE_IO=ssd`
- Placement from the engine log:
- Card 0 runs layers 0-35: expert cache 5,974 slots (17.38 GiB). 12,458 experts are mirrored in pinned host memory (36.28 GiB).
- Card 1 runs layers 36-47: 6,144 of 6,144 slots (18.07 GiB).
- MTP draft layer: 865 MiB.
- VRAM free after load: 1,956 MiB.
- Drafting: Auto picked DFlash2, the engine exited with code 139, and Strata restarted with the MTP (`--spec 2`).
**Host B (1x B70):**
- Setup: same command with `--gpus` left out.
- Engine: `--expert-cache auto --max-context 262144 --kv-resident 32768 --stream-experts --prefill 8192 --kv fp16 --spec 4 --vram-reserve-mib 1536 --draft dflash2 --draft-model data/dflash2/qwen3.8-flash-next-dflash2.q4_0.pack --draft-fallback none --draft-conf 0.6`
- Environment: `ONEAPI_DEVICE_SELECTOR=level_zero:0`, `STRATA_VERIFY_DEVICE_PLAN=1`, `STRATA_VERIFY_NO_HOST=1`, `STRATA_PLE_IO=ssd`
- Placement:
- Expert cache: 7,866 slots (22.93 GiB). 17,496 experts are mirrored in pinned host memory (51.09 GiB).
- DFlash2 drafter: 869 MiB.
- VRAM free after load: 1,371 MiB.
**Local patches** (`patches/v0.2.12/`, applied with `git apply` in this order):
- `01-sycl-path-idempotent`
- `02-param-forms`
- `03-invoke-forms`
- `04-ple-io-env`
- `05-dump-prompts`
- `06-user-args-dedupe`
- `08-drafting-work-path`
- `09-drafting-engine-binary`
</details>
---
#### 1. Standard benchmark (`strata bench run`, protocol `strata-qwen38-thinking-v1-draft.1`)
Both runs ran inside the engine container against a freshly loaded engine. Both have **Status: complete** and **12 of 12** sanity checks.
Command:
```sh
strata bench run --url http://127.0.0.1:8080 --track custom --quantization UD-Q4_K_XL --track-notes "..."
```
Sampling follows the protocol (thinking on, temperature 1.0, top_p 0.95, top_k 20, reasoning_effort xhigh). Values are median (min-max), n=3.
**Host A, 2x B60, layer split, MTP, `--pipeline-windows 0`** (run `20261108T094336Z-4db71d6b`)
| Case | Prompt tok/s | Decode tok/s | TTFT s | First answer s | Total s | Capped |
| --- | --- | --- | --- | --- | --- | --- |
| short | n/a | 31.4 (28.1-31.6) | 1.89 (1.85-1.90) | 4.85 (3.85-4.98) | 11.6 (10.1-14.0) | 0/3 |
| prompt-4k | 619.5 (593.3-621.9) | 29.7 (28.3-29.8) | 6.66 (6.63-6.95) | 16.82 (16.45-18.86) | 23.8 (23.7-25.0) | 3/3 |
| prompt-16k | 1017.7 (1007.8-1019.0) | 28.8 (28.6-29.8) | 16.16 (16.14-16.31) | 33.21 (n=1) | 34.0 (33.3-34.0) | 3/3 |
**Host B, 1x B70, DFlash2, ZFS dataset `recordsize=16K`, `primarycache=all`** (run `20261108T121816Z-79106905`)
| Case | Prompt tok/s | Decode tok/s | TTFT s | First answer s | Total s | Capped |
| --- | --- | --- | --- | --- | --- | --- |
| short | n/a | 45.8 (40.5-48.6) | 0.72 (0.71-0.74) | 2.47 (2.27-3.66) | 7.1 (6.4-7.6) | 0/3 |
| prompt-4k | 1370.1 (1336.1-1370.3) | 43.6 (39.4-46.0) | 3.02 (3.02-3.10) | 8.76 (8.19-9.32, n=2) | 14.7 (14.2-16.0) | 3/3 |
| prompt-16k | 1728.1 (1421.3-1728.4) | 37.8 (37.6-44.8) | 9.52 (9.52-11.57) | 16.06 (n=1) | 23.0 (23.0-23.1) | 3/3 |
The same host before the ZFS change (`recordsize=1M`, `primarycache=metadata`, run `20261108T094605Z-df49e469`):
- prompt-4k: 1400.1 (847.0-1403.2) tok/s;
- prompt-16k: 1544.9 (985.9-1553.8) tok/s;
- decode: 44.8 / 42.8 / 37.3 tok/s.
The model files hash the same in both runs.
"Capped" means the request reached the protocol's 512-token cap, counting reasoning and answer together. With reasoning_effort xhigh this is expected for the 4K/16K cases and is not a failure. The `results.json`, `outputs.jsonl` and README folders are available on request.
---
#### 2. Context sweep (style of #5138)
Method:
- Each prompt is the same document set (`docs/*.md` of the repo) repeated to length.
- It starts with a unique first line and a different starting point per run, so nothing is reused (the engine reported `0 reused` for every request).
- Greedy decoding (temperature 0, thinking left at the model default), streamed through the OpenAI endpoint. The model writes 300 tokens, and every request reached 300.
- Decode is `(completion tokens - 1) / (last token - first token)`, measured by the client. It agrees with the engine's own figure within 0.1 tok/s.
- Prompt read speed and draft acceptance come from the engine's `strata serve: prompt ... read in ... drafts accepted N of M` lines.
- Two runs per length; both values are shown.
**Host A, 2x B60, layer split, MTP `--spec 2`, `--pipeline-windows 0`**
| Prompt tokens (run 1 / 2) | Prompt read, tok/s | Time to first token, s | Decode, tok/s | Drafts accepted |
|---:|---:|---:|---:|---:|
| 1,012 / 952 | 260 / 255 | 3.9 / 3.8 | 32.5 / 32.3 | 94% / 83% (315/352) |
| 8,275 / 8,572 | 774 / 761 | 10.7 / 11.3 | 35.7 / 37.2 | 86% / 90% (272/310) |
| 35,978 / 35,919 | 1,059 / 1,061 | 34.0 / 33.9 | 33.2 / 35.9 | 85% / 85% (239/281) |
| 67,927 / 67,850 | 1,113 / 1,114 | 61.1 / 61.0 | 33.4 / 34.9 | 81% / 82% (245/301) |
| 128,685 / 128,796 | 1,116 / 1,115 | 115.4 / 115.7 | 33.1 / 31.7 | 82% / 88% (253/299) |
| 207,262 / 208,168 | 1,080 / 1,079 | 192.2 / 193.0 | 29.4 / 31.9 | 88% / 87% (264/302) |
| 259,030 / 257,696 | 1,058 / 1,057 | 245.0 / 243.9 | 31.2 / 29.5 | 83% / 81% (264/322) |
The 1K-128K rows and the 200K-259K rows come from two engine sessions with identical settings: the engine was restarted for the standard run in between.
**Host B, 1x B70, DFlash2 `--spec 4`, ZFS dataset `recordsize=16K`, `primarycache=all`**
| Prompt tokens (run 1 / 2) | Prompt read, tok/s | Time to first token, s | Decode, tok/s | Drafts accepted |
|---:|---:|---:|---:|---:|
| 1,012 / 956 | 628 / 609 | 1.6 / 1.6 | 47.0 / 40.6 | 89% / 86% (339/386) |
| 8,275 / 8,568 | 1,422 / 1,303 | 5.9 / 6.6 | 39.9 / 42.9 | 85% / 86% (333/390) |
| 35,984 / 35,919 | 1,527 / 1,554 | 23.6 / 23.2 | 38.9 / 40.5 | 81% / 82% (275/337) |
| 67,927 / 67,853 | 1,550 / 924 | 43.9 / 73.5 | 38.2 / 37.9 | 76% / 79% (299/387) |
| 128,688 / 128,810 | 1,531 / 1,525 | 84.2 / 84.6 | 36.1 / 36.4 | 81% / 71% (271/359) |
| 207,264 / 208,163 | 1,458 / 1,456 | 142.3 / 143.2 | 37.3 / 38.4 | 84% / 81% (367/445) |
| 259,030 / 257,694 | 1,429 / 1,425 | 181.5 / 181.0 | 42.3 / 35.6 | 82% / 80% (360/444) |
Observations:
- **Decode holds at long context on both hosts.**
- B70 with DFlash2: 36-42 tok/s at 207K-259K.
- 2x B60 with MTP: 29-32 tok/s.
- **Draft acceptance with greedy decoding:**
- MTP on 2x B60: 81-94%.
- DFlash2 on B70: 71-89%.
- **2x B60 prompt reading is flat from 32K to 259K** (1,057-1,116 tok/s), even though card 0 reads 12,458 experts from host memory over PCIe 4.0 x8 and the RAM runs at 3,600 MT/s.
- **B70 prompt reading is flat from 36K to 259K** (1,425-1,554 tok/s), above the 1,267-1,312 tok/s at 140K-235K in #5138. One outlier remains: 68K run 2 at 924 tok/s.
---
#### 3. Long-prompt reading depends onEn el sitio
Enlaces a install, modelos, releases.