#6365 [Engine] Prefer FP32 TLA sparse prefill on Battlemage in builds with local TLA headers

open · @xiangyuT · 0 commentaires · Sur GitHub

BenchmarksSetup & installServer & APINVIDIA / CUDAModels & quantsDocumentationWindowsLinux

Description

## Problem and result

The default prompt attention for INT8 KV on Battlemage is the INT8 ESIMD kernel. On B70 it misses the FP64 bound of `qsa_prompt_attn_parity` at all six tested contexts, by 150–180x the warp kernel's error. With the sycl-tla headers, this PR makes supported sparse prefill on Battlemage prefer the FP32 TLA kernel instead. TLA's output is byte-identical to the FP32 warp reference. Builds without the headers keep ESIMD; #6366 proposes adding the headers as a submodule.

**Precision on B70.** ESIMD fails at every context. Per-cell, XMX, TLA, and ESIMD with Q and P at three int8 levels (`STRATA_ESIMD_LV=3`) all pass. Largest absolute error against sampled FP64, 16 query positions per row, INT8 KV. A kernel passes when two things hold: its error is within the row's bound, max(4 × the warp kernel's error, 1e-6 × output scale), and its output is within 1e-4 of the output scale of the warp kernel's output. That is the unchanged rule of `qsa_prompt_attn_parity`, with the FP32 warp kernel as the reference. Raising only P or only Q to three levels (`STRATA_ESIMD_LV=23` or `32`) still fails all six contexts (1.6e-4 to 3.6e-4). TLA's output hash equals warp's at all six. ✅ passes, ❌ fails.

| Context | Queries | Bound | ESIMD | ESIMD LV 3 | Per-cell | XMX | TLA | Warp FP32 (reference) |
|---|---:|---:|---:|---:|---:|---:|---:|---:|
| 1,024 | 512 | 9.2e-06 | ❌ 4.18e-04 | ✅ 2.22e-06 | ✅ 2.57e-06 | ✅ 1.65e-06 | ✅ 2.30e-06 | 2.30e-06 |
| 1,500 | 512 | 7.0e-06 | ❌ 2.91e-04 | ✅ 2.67e-06 | ✅ 2.59e-06 | ✅ 1.90e-06 | ✅ 1.76e-06 | 1.76e-06 |
| 2,100 | 256 | 8.1e-06 | ❌ 3.19e-04 | ✅ 2.80e-06 | ✅ 3.16e-06 | ✅ 2.00e-06 | ✅ 2.03e-06 | 2.03e-06 |
| 4,096 | 512 | 9.6e-06 | ❌ 3.60e-04 | ✅ 2.45e-06 | ✅ 2.36e-06 | ✅ 2.15e-06 | ✅ 2.39e-06 | 2.39e-06 |
| 8,192 | 512 | 7.0e-06 | ❌ 3.09e-04 | ✅ 2.76e-06 | ✅ 2.59e-06 | ✅ 2.04e-06 | ✅ 1.75e-06 | 1.75e-06 |
| 32,768 | 512 | 7.1e-06 | ❌ 2.87e-04 | ✅ 2.91e-06 | ✅ 3.38e-06 | ✅ 2.82e-06 | ✅ 1.77e-06 | 1.77e-06 |
| **Passed** | | | **0/6** | **6/6** | **6/6** | **6/6** | **6/6** | — |

**Speed.** TLA is the fastest of the kernels that pass, at every measured length.
- Against ESIMD at three levels, TLA is 24–44% faster in the attention call. In complete prefill it is 1.8% faster at 2K and 5.8% faster at 32K.
- Against default ESIMD, complete prefill is +0.15% at 1K and +0.54% at 2K. At 4K and 32K TLA is faster, by 0.9% to 3.1%; 8K is within noise.

Complete engine prefill uses one executable, picks the dispatch with environment variables and starts a fresh engine process per arm. Each cell is the mean of three requests with 256 generated tokens each, all of which were checked. Two ESIMD arms bracket the alternatives, and "faster pairs" compares each request with the mean of the two.

| Input | ESIMD s | Per-cell s (Δ) | XMX s (Δ) | TLA s (Δ, faster pairs) | ESIMD drift |
|---|---:|---:|---:|---:|---:|
| 1,024 | 1.010 | 1.039 (+2.82%) | 1.143 (+13.17%) | 1.012 (+0.15%, 1/3) | -0.03% |
| 4,096 | 2.412 | 2.583 (+7.13%) | 3.872 (+60.57%) | 2.390 (-0.88%, 3/3) | -0.02% |
| 8,192 | 6.340 | 6.628 (+4.54%) | 9.714 (+53.21%) | 6.235 (-1.65%, 3/3; median pair -0.32%) | +1.10% |
| 32,768 | 22.137 | 23.581 (+6.52%) | 37.141 (+67.78%) | 21.764 (-1.69%, 3/3) | +0.22% |

At 8K one TLA request (6045.3 ms) carries most of the mean; the other two pairs are -0.24% and -0.32%, within the control drift.

A second bracket added 2K and precise ESIMD, on the same host with the same executable and settings but another B70. Its arms ran in this order: ESIMD, TLA, ESIMD with `STRATA_ESIMD_LV=3`, ESIMD.

| Input | ESIMD s | ESIMD LV 3 s (Δ, faster pairs) | TLA s (Δ, faster pairs) | TLA vs LV 3 | ESIMD drift |
|---|---:|---:|---:|---:|---:|
| 2,048 | 1.518 | 1.553 (+2.35%, 0/3) | 1.526 (+0.54%, 0/3) | -1.77% | 0.00% |
| 32,768 | 22.349 | 22.989 (+2.87%, 0/3) | 21.647 (-3.14%, 3/3) | -5.84% | -1.28% |

## Related discussion

Issue: #6364, where scope agreement with the maintainer is pending. The B70 failures, the maintainer's request and the retest: [Problem and logs](https://github.com/Niko1221/Strata/issues/6045) · [Maintainer’s request](https://github.com/Niko1221/Strata/issues/6045#issuecomment-6994158015) · [v0.2.16 retest](https://github.com/Niko1221/Strata/issues/6045#issuecomment-6994575466). The retest used a source JIT build of tag v0.2.16 on Debian sid with icpx 2026.2. #6366 follows this PR with a proposal to add sycl-tla as a git submodule, so that default and release builds include TLA.

## Approach

- **Kernel.**
  - Selected keys are shared across the grouped query heads.
  - Query preparation and gated output are fused where possible.
  - The FP32 reduction order is preserved, so the output matches the warp kernel bit for bit.
  - Unsupported devices, pools, shapes or alignment are rejected at the TLA entry and keep the existing batch and per-cell/warp dispatch.
- **Dispatch.**
  - TLA uses the ESIMD kernel's run-time gate: Battlemage through Level Zero. That covers the documented Battlemage AOT targets, `bmg-g21` and `bmg-g31`, and the cards in the setup table: B50, B60, B65, B70, B570 and B580. [Arc support](https://github.com/Niko1221/Strata/blob/58ee3227720629ff2e41d640165ebe89c880dfef/docs/INTEL_ARC.md) · [Release build](https://github.com/Niko1221/Strata/blob/58ee3227720629ff2e41d640165ebe89c880dfef/tools/packaging/build_sycl_linux.sh)
  - `STRATA_SYCL_TLA_ATTN=0` restores the previous dispatch at run time, and the Docker launcher passes it to the container.
- **Startup.** When the prompt attention would prefer TLA, `Prefill::init` builds TLA's ten kernels for its device and logs the time. Device code is split per kernel, so a JIT build otherwise compiled each kernel at its first launch, which can fall in the middle of a prompt.
- **CMake.**
  - `STRATA_SYCL_TLA_DIR` names a local sycl-tla checkout; nothing is fetched.
  - `STRATA_SYCL_TLA_ATTN=AUTO`, the default, re-evaluates the headers and the target on each configure. It builds TLA for portable JIT and for any Battlemage AOT list (`bmg-g21`, `bmg-g31` or both). When a header path is given, it warns about missing headers.
  - `ON` requires the headers and a portable or Battlemage target, and `OFF` leaves TLA out.
  - A build tree configured when TLA was an opt-in BOOL option holds `OFF` in its cache, and a new default does not replace a cached value. Configure moves that value to `AUTO` (`ON` stays `ON`) and says so.
  - Release packaging and CI build portable JIT without a sycl-tla checkout, so they keep ESIMD.
- **Docs and the Docker build.** `docs/INTEL_ARC.md` gets an "FP32 TLA prompt attention" section, and `engine/sycl/tools/build.sh` takes `TLA_DIR` (below).
- **Tests.** `qsa_bmg_default_prefill_parity`, `qsa_bmg_tla_pipeline_parity` and `qsa_bmg_tla_prefix_parity` cover the default dispatch, query preparation, gated output and the key workspace.

**Getting sycl-tla.** TLA is built from [sycl-tla](https://github.com/intel/sycl-tla), Intel's SYCL port of CUTLASS and CuTe. It is header-only and tested at commit `87f6850680a580654b9ea2c80dbc01aeb36ad231`. The build reads only its `include/` folder; nothing of sycl-tla is built or installed. The new "FP32 TLA prompt attention" section of `docs/INTEL_ARC.md` gives the steps:

```sh
# the headers at the tested commit, next to the checkout (or the archive: https://github.com/intel/sycl-tla/archive/87f6850680a580654b9ea2c80dbc01aeb36ad231.tar.gz)
git clone https://github.com/intel/sycl-tla.git ../sycl-tla
git -C ../sycl-tla checkout 87f6850680a580654b9ea2c80dbc01aeb36ad231
# a build without Docker
cmake -S engine/sycl -B build-sycl -G Ninja -DCMAKE_C_COMPILER=icx -DCMAKE_CXX_COMPILER=icpx -DSTRATA_SYCL_TLA_DIR=$PWD/../sycl-tla
# the Docker build: the folder above the checkout is /work in the container
docker run --rm -u $(id -u):$(id -g) -v "$PWD/..:/work" -e AOT=bmg-g31 -e REPO=/work/$H -e TLA_DIR=/work/sycl-tla \
    -e BUILD_DIR=/work/$H/build-sycl-aot strata-sycl-dev "cd /work/$H && bash engine/sycl/tools/build.sh strata"
```

The configure output says `Strata SYCL: Battlemage TLA prompt attention: ON` when TLA is built. `engine/sycl/tools/build.sh` passes `TLA_DIR` to CMake, also for an existing build folder. #6366 proposes replacing the manual checkout with a git submodule at `engine/third_party/sycl-tla`.

## Validation

Base `3142b66b96e9c0e775e30f620b11b191a1427c2a` (`main` after 0.2.17), head `86311130aee3873444f062bcf5d9c31094d9fa38` (three commits). The PR was rebased from `58ee3227720629ff2e41d640165ebe89c880dfef`; only the generated settings register and `docs/SETTINGS.md` needed resolving, and the code changes are unchanged. After the rebase, the head was rebuilt for `bmg-g31` with local headers, and on B70 the three `qsa_bmg_*` CTests passed. The legacy CTest still fails only its three ESIMD cases. The precision, speed and token data below were measured on base `58ee3227`, with the g31 AOT engine of an earlier revision of this PR (SHA-256 `c6c0c801d157de650704c19a85b49bb3a00a824bb9d8eebb75312b41b2410726`), whose run-time gate selected only the B70 device id. The head gates TLA on every Battlemage GPU and leaves the B70 path unchanged; the check is under Passed. The last two commits follow the B70 report in this PR's comments: `09c25834` moves a build tree's cached opt-in `OFF` to `AUTO`, and `86311130` builds the TLA kernels at startup. Their checks are the two "Follow-up" items under Passed.

| Item | Setting |
|---|---|
| GPU | Arc Pro B70: Xe2 (Battlemage), PCI ID `8086:e223`, 32,656 MiB reported by xpu-smi. Two such cards in one host: one ran Run 1 and the first precision and operator tables; the other ran the second bracket, the ESIMD level measurements, the check of the head and the follow-up |
| Host | Ubuntu 24.04.2 LTS, Linux 6.17.0-1007-intel; Xeon w5-3435X, 1 socket, 16 cores / 32 threads |
| Container | Ubuntu 24.04.5 LTS; oneAPI icpx 2026.1.1; compute runtime and ocloc 26.31.39395.16; Level Zero loader 1.32.0; 16-CPU quota, 128 GiB memory limit |
| Engine placement | `taskset -c 0-10,12-15` and `--host-core last`: 14 pool workers on CPUs 0–10 and 12–14, host thread on CPU 15; 12 stager threads (`STRATA_STAGER_THREADS=12`) |
| Model, settings | Flash-Next GSQ-RCO IQ2_XS, INT8 KV, MTP spec 4, PCIe share 0.29 |
| Requests | One client; every prompt fresh (no prefix reuse); every reply 256 generated tokens |

**Passed:**

- **Builds.** Built from the head's engine sources with local headers:

  | Target | `AUTO` result | Built | Engine bytes |
  |---|---|---|---:|
  | g21 | TLA on | `strata`, `qsa_prompt_attn_parity` | 114,337,368 |
  | g31 | TLA on | `strata`, `qsa_prompt_attn_parity` | 114,337,368 |
  | mixed | TLA on | configured | — |
  | jit | TLA on | `qsa_prompt_attn_parity` | — |
  | plain (no headers) | TLA off | configured | — |

- **CTests.** On B70 the three `qsa_bmg_*` CTests passed in the g31 and JIT builds.
- **B70 path unchanged.** A fresh engine from the head's g31 build served the three measured 2K requests after three warmups. It reproduced the token sequences of the earlier TLA arm exactly, at 1528.1, 1527.1 and 1522.6 ms prefill against the earlier 1527.9, 1526.7 and 1522.3 ms.
- **Docker build script.** `engine/sycl/tools/build.sh` was checked three ways:
  - without `TLA_DIR` it configured no TLA object;
  - given `TLA_DIR`, it reconfigured an existing build folder and added the TLA object;
  - a fresh configure with `TLA_DIR` reported TLA on.
- **sycl-tla commit.** The tested commit is on sycl-tla's `main`, and its archive unpacks to `sycl-tla-87f6850680a580654b9ea2c80dbc01aeb36ad231`.
- **Reconfiguration.** On an earlier revision with the same header checks, thirteen reconfigurations covered both AOT list orders, cache reuse, invalid local headers and forced ON/OFF. Two partial-header trees each named only the missing header in the warning.
- **Other checks.** The host wrapper and settings checks passed.
- **Follow-up: kernel compiles in the middle of a prompt.** The B70 report in this PR's comment

Sur le site

Liens install, modèles, releases.