#6567 [Engine] Group routed projections and reuse expert weights for prefill in the SYCL engine on Battlemage (BMG)

open · draft · @xiangyuT · 0 comentários · No GitHub

Server & APIMulti-GPUModels & quantsLinux

Descrição

## Problem and result

A streamed SYCL prefill can fetch and expand the same expert weights again in the next chunk. This PR groups compatible routed projections automatically on Battlemage and reuses weights when adjacent chunks can share them. On one Arc Pro B70, 32K engine prefill fell from **22.058 to 16.307 s (-26.07%)**. Client TTFT went from 22.096 to 16.345 s and TPOT from 11.532 to 11.539 ms/token (+0.06%).

Both runs left `STRATA_LEND_MIRROR` unset. The engine mirrored 1,200 borrowed slots. Neither `STRATA_SYCL_BMG_ROUTED_EXPERTS` nor `STRATA_SYCL_BMG_EXPERT_WINDOW` was set. Local expert TLA headers were available for the PR build.

| Input tokens | Clean prefill (s) | PR prefill (s) | Prefill change | Client TTFT (s), clean → PR | TPOT (ms/token), clean → PR | TPOT change | Equal 256-token replies |
|---:|---:|---:|---:|---:|---:|---:|---:|
| 1,024 | 0.951 | 0.879 | -7.57% | 0.973 → 0.901 | 11.423 → 11.412 | -0.10% | 3/3 |
| 4,096 | 2.339 | 2.131 | -8.90% | 2.363 → 2.155 | 10.985 → 11.048 | +0.57% | 3/3 |
| 8,192 | 6.091 | 4.583 | -24.77% | 6.117 → 4.608 | 12.343 → 12.318 | -0.20% | 3/3 |
| 14,192 | 10.034 | 7.511 | -25.14% | 10.063 → 7.540 | 12.010 → 11.999 | -0.10% | 3/3 |
| 27,000 | 19.226 | 14.350 | -25.36% | 19.262 → 14.385 | 11.810 → 11.791 | -0.16% | 3/3 |
| 30,000 | 20.730 | 15.148 | -26.93% | 20.767 → 15.185 | 12.047 → 12.014 | -0.27% | 3/3 |
| 32,768 | 22.058 | 16.307 | -26.07% | 22.096 → 16.345 | 11.532 → 11.539 | +0.06% | 3/3 |

Each arm used one process with three excluded warmups and three distinct fresh requests at each of 1K, 4K, 8K and 32K, then three fresh requests per window-boundary length. All measured prompts produced 256 tokens. TTFT measures the time to the first nonempty response text. TPOT is the time between the first and last nonempty response text divided by 255.

<details><summary>Individual measured requests and within-process variation</summary>

The three entries follow the same input order. The number in parentheses is the maximum minus the minimum.

| Run | Input tokens | Engine prefill (s) | TTFT (s) | TPOT (ms/token) | Total (s) |
|---|---:|---|---|---|---|
| Clean | 1,024 | 0.951 / 0.953 / 0.950 (0.003) | 0.973 / 0.975 / 0.972 (0.003) | 11.639 / 10.966 / 11.665 (0.700) | 3.942 / 3.772 / 3.948 (0.175) |
| PR | 1,024 | 0.880 / 0.879 / 0.879 (0.001) | 0.902 / 0.901 / 0.901 (0.001) | 11.626 / 10.956 / 11.653 (0.698) | 3.868 / 3.696 / 3.874 (0.178) |
| Clean | 4,096 | 2.339 / 2.339 / 2.339 (0.001) | 2.363 / 2.363 / 2.363 (0.000) | 11.157 / 10.890 / 10.908 (0.268) | 5.209 / 5.140 / 5.146 (0.069) |
| PR | 4,096 | 2.131 / 2.132 / 2.130 (0.002) | 2.156 / 2.156 / 2.154 (0.002) | 11.243 / 10.851 / 11.049 (0.392) | 5.024 / 4.923 / 4.972 (0.101) |
| Clean | 8,192 | 6.054 / 6.096 / 6.124 (0.069) | 6.080 / 6.122 / 6.149 (0.069) | 12.292 / 12.433 / 12.304 (0.141) | 9.216 / 9.294 / 9.288 (0.078) |
| PR | 8,192 | 4.563 / 4.600 / 4.585 (0.037) | 4.589 / 4.626 / 4.610 (0.037) | 12.198 / 12.479 / 12.278 (0.281) | 7.701 / 7.809 / 7.743 (0.109) |
| Clean | 14,192 | 10.467 / 9.921 / 9.714 (0.753) | 10.496 / 9.950 / 9.743 (0.753) | 12.103 / 11.785 / 12.143 (0.357) | 13.584 / 12.956 / 12.841 (0.743) |
| PR | 14,192 | 7.368 / 7.539 / 7.627 (0.260) | 7.397 / 7.568 / 7.656 (0.259) | 12.095 / 11.767 / 12.134 (0.368) | 10.482 / 10.570 / 10.751 (0.269) |
| Clean | 27,000 | 19.076 / 19.584 / 19.018 (0.565) | 19.111 / 19.619 / 19.055 (0.564) | 11.891 / 11.878 / 11.662 (0.229) | 22.144 / 22.649 / 22.030 (0.619) |
| PR | 27,000 | 14.212 / 14.588 / 14.248 (0.377) | 14.247 / 14.623 / 14.285 (0.376) | 11.883 / 11.855 / 11.636 (0.247) | 17.278 / 17.647 / 17.253 (0.394) |
| Clean | 30,000 | 21.001 / 20.622 / 20.568 (0.433) | 21.038 / 20.659 / 20.605 (0.433) | 11.927 / 12.283 / 11.931 (0.356) | 24.081 / 23.792 / 23.649 (0.432) |
| PR | 30,000 | 15.050 / 15.092 / 15.303 (0.253) | 15.086 / 15.130 / 15.340 (0.254) | 11.925 / 12.206 / 11.912 (0.295) | 18.128 / 18.244 / 18.379 (0.251) |
| Clean | 32,768 | 22.114 / 21.794 / 22.267 (0.473) | 22.153 / 21.832 / 22.304 (0.472) | 11.628 / 11.198 / 11.768 (0.570) | 25.119 / 24.689 / 25.306 (0.617) |
| PR | 32,768 | 16.508 / 16.369 / 16.043 (0.464) | 16.545 / 16.407 / 16.082 (0.463) | 11.657 / 11.114 / 11.845 (0.731) | 19.519 / 19.242 / 19.104 (0.415) |

</details>

## Related discussion

Maintainer scope agreement for the proposed automatic Battlemage dispatch is pending.

## Approach

The SYCL engine selects the grouped kernels for compatible Level Zero Battlemage inputs. Set `STRATA_SYCL_BMG_ROUTED_EXPERTS=0` to restore the previous routed path and disable the window. Set only `STRATA_SYCL_BMG_EXPERT_WINDOW=0` to keep grouped projections and use ordinary chunk traversal.

At build time, `STRATA_SYCL_TLA_EXPERT_GU=AUTO` uses available local sycl-tla headers on a compatible target. Without headers the ordinary grouped kernels remain available. The window requires a build with expert TLA, a single unbatched in-order queue, a streamed 48-layer native model with 512 experts, top-10 routing and widths 2,560/640, plus IQ2_XXS/IQ2_S/IQ1_M gate/up and Q2_0 down. The 8,192-token plan pairs chunks when the second has at least 4,096 rows and checkpoint boundaries allow it. Both the workspace estimate and its allocation check the same model layout.

Earlier checks on these same SYCL sources exercised an explicit window opt-out, a disabled routed master with the window requested, a build without local expert TLA headers, and rejection of split/batched and out-of-order sessions. The default-configuration timings here use the submitted commit with local TLA headers.

## Validation

**Passed:**

- Built the unmodified [upstream base](https://github.com/Niko1221/Strata/commit/6e38f588a646c0ee6948a74afbe670eb75896106) and the submitted single commit `0cee0ff05890fb171bbc813433391821e818beaf` in Release mode for BMG G21/G31. The performance run used Git tree `2cd66bf67cf1a85374c42e1c489429b1cad0fea2`, identical to the tree of the submitted commit.
- Matched all 256 token IDs for each of the 21 measured prompts against clean main. At the measured head, the production engine logged use of two adjacent chunks per expert window with the runtime controls unset. Earlier tests of the same SYCL sources covered grouped oneMKL byte parity and the explicit rollback paths. Only the end-to-end results above are new measurements.

```sh
CC=icx CXX=icpx cmake -G Ninja -S engine/sycl -B build -DCMAKE_BUILD_TYPE=Release \
  -DSTRATA_SYCL_AOT=bmg-g21,bmg-g31 -DSTRATA_SYCL_PARITY=ON -DSTRATA_SYCL_TLA_ATTN=OFF -DSTRATA_SYCL_TLA_EXPERT_GU=AUTO -DSTRATA_SYCL_TLA_DIR=/path/to/sycl-tla
cmake --build build --target strata
```

| Configuration | Values for both arms |
|---|---|
| Hardware and software | One B70 (G31) with 32 GB, Xeon w5-3435X, 16 cores / 32 threads. OMIX 0.4.1, Ubuntu 24.04.5, Linux 6.17.0-1007-intel, icpx 2026.1.1, oneMKL 2026.1.0, compute runtime and ocloc 26.31.39395.16. Container limits were 16 CPUs and 128 GiB RAM. |
| Model and engine | One unbatched client, streamed Qwen3.8-Flash-Next GSQ-RCO IQ2_XS, INT8 KV, MTP spec 4, ESIMD prompt attention. |
| Launcher | `STRATA_STAGER_THREADS=12`, `STRATA_VERIFY_DEVICE_PLAN=1`, `STRATA_VERIFY_NO_HOST=1`, matching the [upstream SYCL launcher](https://github.com/Niko1221/Strata/blob/6e38f588a646c0ee6948a74afbe670eb75896106/engine/sycl/serve/strata-sycl.sh#L31). |
| Common engine arguments | `--expert-cache 17558 --stream-experts --vram-reserve-mib 2048 --prefill auto --spec 4 --spec-min-p 0.5 --max-context 65536 --kv int8 --kv-resident 32768 --pcie-frac 0.29` |

**Failed:**

None in the reported builds and requests.

**Not run:**

G21 hardware, other model layouts, the full SYCL CTest suite, and other backends on this rebase.

## Limits

The performance measurements cover one active B70 with this streamed model and mirror configuration. The expert TLA build used [Intel sycl-tla](https://github.com/intel/sycl-tla/tree/87f6850680a580654b9ea2c80dbc01aeb36ad231). Multi-GPU split and batched inference were outside this comparison.

No site

Links install, modelos, releases.