#5759 [Benchmark] Arc Pro B70 (x16) + B60 (x8) layer split on EPYC 7443: IQ3_S/IQ2_XS at 262K, engines 0.1.54-0.1.59.1

closed · @Fett2 · 1 commentaires · Sur GitHub

BenchmarksSetup & installServer & APIModels & quantsDocumentationWindowsLinux

Description


### What this box is, and what it is good for measuring

A permanent production service on a server-class host: single-socket EPYC 7443 (Zen 3 Milan, 24 physical / 48 logical cores, **no AVX-512**, single NUMA node, 8-channel DDR4-3200 with the expert files fully page-cached), two Battlemage cards on **deliberately asymmetric PCIe paths** - Arc Pro B70 32 GB at Gen4 x16 and Arc Pro B60 24 GB at Gen4 x8 - running one Qwen3.8-Flash-Next model as a layer split at `--context 262144` under systemd, serving real agent traffic (this operator's own chat/agent sessions ride it) plus measured blocks. Nobody else in the reports here mixes a B70 with a B60, and nobody else reports a machine where the model files never leave the OS page cache.

I keep an embed and reranker model on the B60 always, and all tests were run with those models loaded and idle with no traffic. Verified before and after each test that they nothing.

Everything below comes from one record kept on the box (§-numbered below), with house hygiene: **every pass starts from a fresh engine restart; no chat or RAG traffic runs inside a measured start; two passes per condition, and they must agree**; the pair of vLLM containers (embed/rerank, a *different* service on 8001/8002) sits loaded and idle as an interference witness with `vllm:request_success_total` 0/0 across every block. Noise: decode repeat-spread up to ~8% at this depth is noise here; prefill is far steadier (probe CV <0.5%, fixture passes within 1-2%). Percentages marked *(derived)* are computed from the medians, not measured directly.

### Hardware and software (current state; history noted)

| | |
|---|---|
| CPU / RAM | EPYC 7443, 8-ch DDR4-3200; grew 96 GB to **128 GB installed** (`MemTotal` 123 GiB) - the growth is itself a measured arm below |
| GPU 0 (stage 0) | Arc Pro B60 24 GB (`e211`, usable 23.875 GiB), root port Gen4 **x8**, engine probe **14.2 GB/s** host->device |
| GPU 1 (stage 1) | Arc Pro B70 32 GB (`e223`, usable 31.89 GiB), root port Gen4 **x16**, engine probe **28.4-28.7 GB/s** |
| Quirk for host-side records (#4610) | these Battlemage cards' own endpoint registers read `Gen1 x1` **whatever the link runs at**; only root/switch-port reads and the engine's probes tell the truth. On this box both paths run Gen4 today; before a reseat they were both x4 (`PCIe probe 7.2 GB/s` era) - the probes moved to 14.2 / 28.7 with no config change |
| OS / kernel | Ubuntu-family, Linux 7.3.0-x, Python 3.14.8 |
| Engine | built from tags in the `strata-sycl-dev` container (oneAPI **2026.1**, AOT `bmg-g31`); versions walked 0.1.54 -> 0.1.55 -> 0.1.56 -> 0.1.58 -> **0.1.59.1** (repo v0.2.13.1); rollbacks kept beside the binary and every build verified by stamp (`strings | grep engine=`), not by exit code |
| Model | Qwen3.8-Flash-Next **GSQ-RCO IQ3_S** (native experts: repo pack + GGUF shards), and an **IQ2_XS** leg (§4) |
| Config (steady state) | `gpu [1,0]`, `layer_split "14"` pinned (k14), expert caches **7,168 + 12,110 slots** (23.92 GiB on the B70; ~19.3k resident of the model's 24,576 experts - total per #5160's report, our logs confirm only the two slot counts), `--stream-experts`, `--kv int8`, `--prompt-cache 6`, `--conversation-cache-mib 24576 --conversation-cache-slots 8`, `--vram-reserve-mib 2048`, MTP pinned (`drafting: "mtp"`, `--spec 4 --spec-min-p 0.5`), pipeline windows default (2 on Arc B), `STRATA_VERIFY_NO_HOST=0` (pinned off - the streaming path makes `NO_HOST=1` invalid here; see #5747/#5160), serve repetition guard `repetition_penalty 1.05, penalty_last_n 32` (cost measured: ~8% decode with the full window, inside overlap at L32) |

### Speed by engine era - block fixtures (9.4K / 41.6K / 128K prompts, fresh restart, quiet)

`fbig` (128K) self-parks at 32,768 reused after its first pass, so its figures are **read-rate** on the new tokens. Two passes each; both quoted when they differed beyond rounding.

| leg | f2 prefill | f1 prefill | fbig read-rate | decode f2 / f1 / fbig |
|---|---|---|---|---|
| 0.1.56, 96 GB (§43) | 395.0 / 392.2 | 1,708.2 / 1,702.0 | - | (older guard state) |
| **RAM 96->128 GB** (§47, same engine) | 406.7 / 407.8 | **1,826.7 / 1,823.0** (+6.9/+7.1% *(derived)* vs §43) | 2,037.3 / 2,041.5 | 53.4/50.3 · 54.2/48.8 · 46.4/47.4 |
| 0.1.58, PCIe pins held (§50) | 404.0 / 406.0 | 1,820.6 / 1,825.3 | 2,038.2 / 2,040.7 | 53.2/46.1 · 49.7/48.5 · 47.1/45.8 |
| 0.1.58, pins removed - defaults (§52) | 404.8 / 405.8 | 1,815.9 / 1,815.0 | 2,021.0 / 2,022.7 | 53.8/51.1 · 52.4/50.6 · 46.1/49.3 |
| **0.1.59.1** (§53, current) | 397.5 / 399.6 | 1,764.2 / 1,758.2 | 1,937.5 / 1,938.7 | 50.2/47.9 · 46.9/50.4 · 48.2/51.8 |

Verdicts measured here: **0.1.56->0.1.58 moved nothing** (every delta inside band). On **0.1.59** all three prefill legs sit 1.5-4.1% lower *in both passes, same direction* - each single delta inside the band, so flagged as a tendency, not claimed as a regression; decode unchanged. The RAM growth bought +3-7% prefill at 41.6K *(derived)* and nothing else. The §46 PCIe pins (`STRATA_PCIE_SHARE_MAX/K_RULE/AUTO_SPLIT`) were tested same-engine against the 0.1.58 defaults (§52): identical inside band, and the auto rule chose **the same split (14)** on its own - pins deleted permanently from our config.

### BetterBench legs (harness 0.6.1, corpus v1.0 `e332dceff176d033`, temp 0.7 / top_p 0.95 / top_k 20, warmup 3, 8 categories x 20, prefill 5 depths x 8, concurrency 1/2/4/8 x 48)

**IQ3_S k14 on 0.1.56 (§44)** - the anchor: per-category decode medians 58.8-78.8 (all-sample median 71.4), prefill 739 / 1,200 / 1,452 / 1,828 / **2,104** tok/s at 1.6K/6K/12K/24K/47K actual prompt tokens, single-stream TTFT p50 1,057-1,138 ms, aggregate tok/s flat with L8 (56.2/56.2/56.2/54.9 - **no batching throughput gain up to 8 concurrent**, though 20-24 of 48 runs batch), arrival p50 @ L2/L4/L8 = 7.6 / 13.8 / **30.0 s**.

**k14 vs k18 split (§45, same day pair):** decode/TTFT/aggregate all tie; **k18 wins prefill at >=24K (+6.2%/+8.8%)** and loses nothing else except L4 arrival shape; default stays k14, with k18 recorded as the measured +6-9% choice for a 32-128K-prompt workload.

**IQ2_XS at the same k14 and near-same engine era (0.1.59.1), two passes, second in strict quiet (§54):**

| measure | §44 IQ3_S (0.1.56) | IQ2_XS p1 | IQ2_XS p2 quiet |
|---|---|---|---|
| decode all-sample median | 71.4 | 78.1 | **77.9** (+9% *(derived)* over §44, both passes) |
| prefill 1.6K/6K/12K/24K/47K | 739/1,200/1,452/1,828/2,104 | 890/1,448/1,737/2,184/2,513 | **903/1,468/1,756/2,197/2,519** (+19-21% at *every* depth) |
| single-stream TTFT p50 | 1,057-1,138 ms | 754-835 | **756-836** (-25 to -28%) |
| aggregate @ L1/2/4/8 | 56.2/56.2/56.2/54.9 | 66.1/66.1/66.8/65.7 | **66.2/65.1/66.0/66.1** (flat, raised ~+19%) |
| arrival p50 @ L2/4/8 | 7.6/13.8/30.0 s | 6.1/17.1/**20.5** | 5.8/16.5/**30.3** |

The engine-version confound runs *against* these deltas (the 0.1.59 prefill tendency above), so the quant effect is a floor. The one number that did **not** reproduce: pass 1's 20.5 s at L8 - the quiet pass reads 30.3 s, essentially §44's and §45's 30.0 s. **The L8 queue shape (~30 s arrival p50) is a property of this engine's queueing, not of split or quant**; the flag `sample_gate` keeps marking all p99 tails under-sampled at n=20 (treat as hints), consistent with how §44/§45 recorded theirs. `batched_runs` ~17-22 of 48 at every level on both quants, `cached_tokens=0` throughout.

### What this box adds for Intel/SYCL users specifically

- Mixed B70+B60 split works and the asymmetry behaves: the engine probes 14.2 vs ~28.5 GB/s and its auto `pcie_frac` lands at 0.39 / 0.55 accordingly; expert caches then size to 7,168 (B60) + 12,110 (B70) slots.
- Smaller streaming quants move **every read-bound number**: the §54 pair is the cleanest quant-only A/B we can offer (+20% prefill at all depths, +9% decode, -26% TTFT) - consistent with #3150's single-B70 IQ2_XS speed but now at split-262K.
- `STRATA_VERIFY_NO_HOST` on *streaming* installs: pinned 0 here by necessity (the flag's own doc forbids it without full VRAM+mirror coverage); weeks of clean output run with adaptive host swaps fully active - relevant to the #5160 clean/corrupt discussion.
- Engine 0.1.59's deterministic prompt GEMMs: this box now runs bit-repeatable Arc output (the release's headline, confirmed here in the sense that mattered - our BetterBench passes agree to 0.3% without a drift asterisk).
- The DFlash2 arm is **closed by engine capability, not config**: named or Auto df2 on a SYCL split hits the fork guard (`sycl/src/program/generate.cpp:4425`) and dies with exit 139 - full report in #5747; the blind form-probe is fixed in PR #5745.

### Not tested here (so nobody reads this as coverage)

No `strata bench run` protocol conformance run yet (our fixtures are in-house: f2/f1/fbig + BetterBench; a standard `--url` run against the live service is queued as a follow-up so this thread can carry a conforming result too). No needle/recall/correctness scores - our service-level checks are output inspection and the repetition guard's log. No vision traffic (encoder off on the measured legs... it is *configured* but no image requests ran). No DFlash2 measurements (capability gap above). No clocks/power readings or card power-limit arms. No driver-stack pinning beyond oneAPI 2026.1 inside the build container - host `intel-*/libze` versions are not recorded at every leg and will not be invented now. No single-card rows (the B60 has never served alone in this record). No `STRATA_SYCL_BF16_DET=0` A/B yet (optional; determinism kept), no long-duration soak claim beyond ~5 weeks with one `!`-loop event (§38, before the sampling guard) and zero since.

### Data provenance

All block fixtures run from one runner (md5 `ff054f22153f5c70969d5c026f4aaff9`, kept on the box), labels like `iq3s-k14-e0158-pin1`, `iq2xs-k14-e0159-p2`; BetterBench run dirs under `~/.betterbench/runs/` carry their own `results.json`. The box's full dated record (54 numbered sections: start-shape greps from the engine log per pass, counters, both passes of every A/B) lives in this operator's log; the runner script, BetterBench `results.json` files and engine start logs can be shared verbatim on request.

Sur le site

Liens install, modèles, releases.