#6364 [Engine] Prefer FP32 TLA over INT8 ESIMD for Battlemage prompt attention

open · @xiangyuT · 0 comentarios · En GitHub

Setup & installAMD / HIPNVIDIA / CUDAModels & quantsDocumentationWindowsLinux

Descripción

### Area

[Engine]

### What do you want to do?

Prefer the FP32 TLA kernel over the INT8 ESIMD kernel for sparse prompt attention on Battlemage GPUs, in builds that have the sycl-tla headers. Builds without them keep ESIMD.

**Problem.** ESIMD is the default prompt attention for INT8 KV on every Battlemage GPU (`intel_gpu_gen == Battlemage`). It quantizes Q and P to two int8 levels (hi + lo).
- Its header says that `qsa_prompt_attn_parity` bounds its difference from FP64. On a B70 it misses that bound at all six tested contexts, by 150–180x the warp kernel's error.
- The B70 reporter reproduced the failures at 1500, 2100 and 8192 on a source JIT build of v0.2.16; disabling ESIMD passed all 12 cases.
- The maintainer asked for a B-series retest, which the reporter supplied.

[Report](https://github.com/Niko1221/Strata/issues/6045) · [Maintainer](https://github.com/Niko1221/Strata/issues/6045#issuecomment-6994158015) · [Retest](https://github.com/Niko1221/Strata/issues/6045#issuecomment-6994575466)

**Precision on B70.** ESIMD fails at every tested context, at its default levels and with only Q or only P raised. Per-cell, XMX and TLA pass, and so does ESIMD with both Q and P at three int8 levels. The table gives the largest absolute error against sampled FP64 (INT8 KV). A kernel passes when two things hold: its error is within the row's bound, max(4 × the warp kernel's error, 1e-6 × output scale), and its output is within 1e-4 of the output scale of the warp kernel's output. That is the unchanged rule of `qsa_prompt_attn_parity`, with the FP32 warp kernel as the reference. `STRATA_ESIMD_LV` sets the int8 levels of Q and P: 2 (2/2, the default), 23 (2/3), 32 (3/2) or 3 (3/3). ✅ passes, ❌ fails.

| Context | Queries | Bound | ESIMD (LV 2, default) | ESIMD LV 23 | ESIMD LV 32 | ESIMD LV 3 | Per-cell | XMX | TLA | Warp FP32 (reference) |
|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|
| 1,024 | 512 | 9.2e-06 | ❌ 4.18e-04 | ❌ 3.58e-04 | ❌ 2.33e-04 | ✅ 2.22e-06 | ✅ 2.57e-06 | ✅ 1.65e-06 | ✅ 2.30e-06 | 2.30e-06 |
| 1,500 | 512 | 7.0e-06 | ❌ 2.91e-04 | ❌ 2.58e-04 | ❌ 1.90e-04 | ✅ 2.67e-06 | ✅ 2.59e-06 | ✅ 1.90e-06 | ✅ 1.76e-06 | 1.76e-06 |
| 2,100 | 256 | 8.1e-06 | ❌ 3.19e-04 | ❌ 3.01e-04 | ❌ 1.61e-04 | ✅ 2.80e-06 | ✅ 3.16e-06 | ✅ 2.00e-06 | ✅ 2.03e-06 | 2.03e-06 |
| 4,096 | 512 | 9.6e-06 | ❌ 3.60e-04 | ❌ 3.44e-04 | ❌ 1.66e-04 | ✅ 2.45e-06 | ✅ 2.36e-06 | ✅ 2.15e-06 | ✅ 2.39e-06 | 2.39e-06 |
| 8,192 | 512 | 7.0e-06 | ❌ 3.09e-04 | ❌ 2.79e-04 | ❌ 1.82e-04 | ✅ 2.76e-06 | ✅ 2.59e-06 | ✅ 2.04e-06 | ✅ 1.75e-06 | 1.75e-06 |
| 32,768 | 512 | 7.1e-06 | ❌ 2.87e-04 | ❌ 2.70e-04 | ❌ 2.33e-04 | ✅ 2.91e-06 | ✅ 3.38e-06 | ✅ 2.82e-06 | ✅ 1.77e-06 | 1.77e-06 |
| **Passed** | | | **0/6** | **0/6** | **0/6** | **6/6** | **6/6** | **6/6** | **6/6** | — |

**Why TLA.** TLA is the fastest of the kernels that pass, at every measured length.
- **Exact.**
  - Its output is byte-identical to the FP32 warp kernel's at all six contexts above and on 5/5 sets of real tensors.
  - The engine generated the same 256 tokens with TLA as with the warp path at 1K, 4K, 8K and 32K.
- **Faster than the other passing kernels.** Against ESIMD at three levels, TLA is 24–44% faster in the attention call. It is 1.8–5.8% faster in complete prefill, measured at 2K and 32K. Per-cell and XMX are slower still.
- **Cost against default ESIMD.**
  - Complete prefill is +0.15% at 1K and +0.54% at 2K, where the attention call is 25–35% slower.
  - At 4K and 32K TLA is faster, by 0.9% to 3.1%. 8K is within noise.

**Builds and devices.** Upstream ships the Linux Intel Arc engine as portable SPIR-V/JIT for Alchemist and Battlemage. Its documented Battlemage AOT targets are `bmg-g21` and `bmg-g31`, and the setup table recognizes B50, B60, B65, B70, B570 and B580. With the headers, AUTO builds TLA for portable JIT and any Battlemage AOT list, and TLA uses the ESIMD kernel's run-time gate. All hardware results here are from B70. [Arc support](https://github.com/Niko1221/Strata/blob/58ee3227720629ff2e41d640165ebe89c880dfef/docs/INTEL_ARC.md) · [Release build](https://github.com/Niko1221/Strata/blob/58ee3227720629ff2e41d640165ebe89c880dfef/tools/packaging/build_sycl_linux.sh)

**Release builds and follow-up.** A build made with the normal release settings has no sycl-tla headers and keeps ESIMD. #6366 proposes a git submodule of intel/sycl-tla at `engine/third_party/sycl-tla`, pinned at the tested commit. Checkouts that have the submodule, the CI SYCL job and release builds would then include TLA.

We propose a submodule rather than a vendored copy because a copy would be hard to keep correct:
- the kernel reads 180 sycl-tla headers, about 7 MB;
- the full list has to be collected from both the host and the device compilation passes;
- the copy would need refreshing on every kernel or pin change.

This would be the repository's first submodule, so how release builds get the headers is the repository owner's decision; #6366 is our proposal. Until it is settled, `docs/INTEL_ARC.md` ("FP32 TLA prompt attention") describes how to get sycl-tla and point the build at it.

**Cost of the alternatives: complete prefill.** Each run uses one executable, picks the dispatch with environment variables and starts a fresh engine process per arm. Each cell is the mean of three measured prompts with 256 generated tokens each. Two ESIMD arms bracket the alternatives, and "faster pairs" compares each prompt with the mean of the two.

Run 1, the four dispatches:

| Input | ESIMD s | Per-cell s (Δ) | XMX s (Δ) | TLA s (Δ, faster pairs) | ESIMD drift |
|---|---:|---:|---:|---:|---:|
| 1,024 | 1.010 | 1.039 (+2.82%) | 1.143 (+13.17%) | 1.012 (+0.15%, 1/3) | -0.03% |
| 4,096 | 2.412 | 2.583 (+7.13%) | 3.872 (+60.57%) | 2.390 (-0.88%, 3/3) | -0.02% |
| 8,192 | 6.340 | 6.628 (+4.54%) | 9.714 (+53.21%) | 6.235 (-1.65%, 3/3; median pair -0.32%) | +1.10% |
| 32,768 | 22.137 | 23.581 (+6.52%) | 37.141 (+67.78%) | 21.764 (-1.69%, 3/3) | +0.22% |

Run 2, 2K and precise ESIMD, on the same host with the same executable and settings but another B70:

| Input | ESIMD s | ESIMD LV 3 s (Δ, faster pairs) | TLA s (Δ, faster pairs) | TLA vs LV 3 | ESIMD drift |
|---|---:|---:|---:|---:|---:|
| 2,048 | 1.518 | 1.553 (+2.35%, 0/3) | 1.526 (+0.54%, 0/3) | -1.77% | 0.00% |
| 32,768 | 22.349 | 22.989 (+2.87%, 0/3) | 21.647 (-3.14%, 3/3) | -5.84% | -1.28% |

<details><summary>Attention-call timings and generated tokens</summary>

Attention calls on frozen real tensors (staged Q, K/V, selection and steps), five queue-profiled windows of 24 calls after warmup. 32K measures the final chunk. The calls exclude query preparation and output gating.

| Prompt | Queries | First position | ESIMD ms | Per-cell ms | XMX ms | TLA ms | Warp ms | TLA Δ | ESIMD drift |
|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|
| 1K | 1017 | 0 | 1.072 | 3.354 | 12.132 | 1.340 | 3.586 | +25.02% | -0.08% |
| 2K | 2041 | 0 | 3.712 | 10.544 | 45.927 | 5.003 | 11.789 | +34.80% | +0.80% |
| 4K | 4089 | 0 | 14.516 | 28.488 | 135.995 | 13.769 | 34.855 | -5.14% | +0.05% |
| 8K | 8185 | 0 | 36.580 | 65.608 | 316.304 | 35.416 | 79.325 | -3.18% | +0.06% |
| 32K | 8185 | 24576 | 46.772 | 79.009 | 362.927 | 39.924 | 97.334 | -14.64% | +0.02% |

Precise ESIMD on the same tensors, in a second session that also re-measured default ESIMD and TLA; default ESIMD drifted at most 1.01%. The last column is the largest absolute difference from the FP32 warp output.

| Prompt | ESIMD ms | ESIMD LV 3 ms | TLA ms | LV 3 vs ESIMD | TLA vs LV 3 | Max diff ESIMD / LV 3 / TLA |
|---|---:|---:|---:|---:|---:|---|
| 1K | 1.072 | 1.908 | 1.342 | +77.1% | -29.7% | 3.8e-4 / 3.8e-6 / 0 |
| 2K | 3.796 | 6.689 | 5.105 | +76.4% | -23.7% | 5.4e-4 / 5.3e-6 / 0 |
| 4K | 14.515 | 23.510 | 13.949 | +62.0% | -40.7% | 5.0e-4 / 5.7e-6 / 0 |
| 8K | 36.510 | 57.701 | 35.992 | +58.0% | -37.6% | 6.0e-4 / 6.4e-6 / 0 |
| 32K | 46.701 | 72.402 | 40.871 | +55.1% | -43.5% | 5.6e-4 / 7.6e-6 / 0 |

Generated tokens:

| Prompt | TLA vs warp reference | TLA vs per-cell: first differing token |
|---|---:|---:|
| 1,024 | 256/256 | 7 |
| 4,096 | 256/256 | 22 |
| 8,192 | 256/256 | 23 |
| 32,768 | 256/256 | 36 |

The warp reference is `STRATA_PROMPT_ATTN_OLD=1 STRATA_ATTN_PERCELL=0`. `OLD` alone selects per-cell; it generated the same 256 tokens as the per-cell dispatch at every length. Per-cell sums in another FP32 order, and such a reordering changes greedy tokens within a few dozen tokens, so only the warp path serves as a token reference.

</details>

No task-level model-quality score was measured.

### What do you do today instead?

On v0.2.16, `STRATA_PROMPT_ATTN_ESIMD=0` avoids the failing path. The legacy test then runs XMX, which its harness enables. A normal model request has XMX off and falls through to per-cell, which is 2.8–7.1% slower in complete prefill. `STRATA_ESIMD_LV=3` also passes the bound and is 2.4–2.9% slower.

With this change, a build with the headers can opt out in three ways:
- leave out the headers;
- configure `-DSTRATA_SYCL_TLA_ATTN=OFF`;
- set `STRATA_SYCL_TLA_ATTN=0` at run time.

Nothing is downloaded at configure or build time.

En el sitio

Enlaces a install, modelos, releases.