Pull requests / #2808
#2808 sycl: prompt attention over 8-query selection unions on XMX (opt-in STRATA_PROMPT_ATTN_UNION=1)
open · @hotschmoe · 0 commentaires · Sur GitHub
BenchmarksMulti-GPUAMD / HIPNVIDIA / CUDAModels & quantsDocumentationWindows
Description
## Title sycl: prompt attention over 8-query selection unions on XMX (opt-in `STRATA_PROMPT_ATTN_UNION=1`) Issue: Resolves #2807 ## Summary On the SYCL port the QSA prompt attention runs the FP32 decode kernel, 32 positions per launch, and every position gathers its own ~2,051 selected K/V cells. The XMX port (`STRATA_PROMPT_ATTN_XMX=1`) is slower than that, and docs/INTEL.md left a kernel over the union of neighbouring positions' selections "not built". This PR adds that kernel, opt-in. For every 8 neighbouring positions a first kernel builds the sorted union of their selected cells, each cell marked with the positions that chose it. The attention kernel dequantizes the INT8 K/V of 64-cell chunks of that union once to FP16 and runs S^T = K Q^T and O += P V on joint_matrix (8x16x16, FP32 sums) for all 96 rows (8 positions x 12 heads). In the online softmax a cell is -inf for the positions that did not select it, so every position attends to exactly the cells it attends to with the FP32 kernel. - Off by default. With the switch unset the outputs and draft counts are identical to upstream main's (both decode runs and a turn with 40,796 tokens of context); the only new code on that path is the switch, read once. - INT8 KV on a Battlemage card. Any other KV format or card, or a failed scratch allocation, keeps the existing path (one stderr line says why). - Not bitwise with the switch set: Q, K, V and the probabilities enter the products as FP16. Measured 2026-10-16 on 2x Intel Arc Pro B70 (32 GB, `xe`, kernel 7.1, Compute Runtime 26.22.38646.4, oneAPI 2026.1, AOT `bmg-g31`), Qwen3.8-Flash-Next ISTA GSQ-RCO IQ3_XXS, `--layer-split 30 --prefill 8192 --max-context 65536 --kv int8 --spec 4 --mtp <BF16 draft layer> --mtp-max-t 4 --spec-min-p 0 --stream-experts --expert-cache auto`, one slot, greedy, fixed prompts; upstream main 296b59d2 against this branch: | | upstream main | `STRATA_PROMPT_ATTN_UNION=1` | | --- | ---: | ---: | | prompt, 4K tokens (two fresh prompts) | 993 / 1,189 tok/s | 1,127 / 1,382 tok/s | | prompt, 32K tokens (two fresh prompts) | 2,004 / 1,956 tok/s | 2,314 / 2,246 tok/s | | first token of a turn with 40,796 tokens of context | 23.28 s | 20.39 s | | decode, 512 tokens (3 runs) | 78.7-81.8 tok/s | 79.1-82.9 tok/s | | one verify window (3 runs) | 39.4-39.7 ms | 39.3-40.0 ms | Decode is not touched. With the switch set the outputs differ (rounding), so decode tok/s moves with the draft acceptance while the verify window takes the same time. Accuracy: - `qsa_prompt_attn_union_parity --selftest` (one card, random INT8 pools behind a shuffled page table, the prompt path's selection layout and cap of 2,051; 1.5K-40K contexts, ragged last groups, pages that are not resident, a bitwise rerun): 5 of 5 pass. Error against FP64 2.6e-3 - 5.0e-3 at an output scale of 3.3-3.6 (FP32 kernel 1.7e-6 - 2.6e-6); union against FP32 on average 7e-5 of the scale. - Teacher-forced NLL (`STRATA_LOGPOS`; 8 chats, each an ~8,400-token context read by the prompt path and a 1,617-token follow-up scored through the decode windows; 12,737 tokens, the same targets in every arm): | arm | mean NLL | top-1 same as switch unset | | --- | ---: | ---: | | switch unset | 2.6234 | - | | `STRATA_PROMPT_ATTN_UNION=1` | 2.5506 | 93.2% | | control: switch unset, `--prefill 4096` (changes only rounding) | 2.6321 | 94.1% | The 0.073 lower NLL is not a measured improvement: single chats move by up to 0.2 nats/token in the control as well (the spread over the 8 chats is 0.12 nats/token in both arms), so 8 chats do not resolve a difference of this size (95% bootstrap interval of the NLL change over the chats: -0.160 to +0.015 for the switch, -0.072 to +0.092 for the control). ## What changed - `sycl/src/kernels/cuda/qsa_prompt_attn_union.dp.cpp` (new): the union kernel (one work-group per 8 positions, 16K-cell local bitmaps), the attention kernel (one work-group per 8 positions and KV head, 12 sub-groups of 16, 256 GRF), the launcher (256 groups per launch pair) and its scratch, one per queue. - `sycl/include/strata/kernels/qsa_prompt_attn_union.hpp` (new): the entry point (the contract of `qsa_prompt_attn_batch`) and the switch. - `sycl/src/kernels/cuda/qsa_prompt_attn.dp.cpp`: the include and the guarded call before the XMX v2 call (4 lines). - `sycl/src/kernels/qsa_prompt_attn_union_parity.cpp` (new): the parity test (`--selftest` in ctest, skip code 77 off Battlemage); `<context> <queries> [reps] [--drift F] [--sel <STRATA_DUMP_SEL file>]` times one chunk. - `sycl/CMakeLists.txt`: the new source in `strata_kernels`, the parity target and its test. - `docs/INTEL.md`: an entry after "XMX v2 in the full matrix, and why a grouped kernel is not next". Left alone: the selection (block scores, top-k), decode, the FP32 and XMX v2 prompt kernels, the other KV formats, and every file outside `sycl/` and `docs/` (CUDA, HIP and Metal builds see no change). ## Extra Notes - Prepared with AI assistance (Claude Code): the code, the measurements and this text were produced by Claude on the submitter's two Arc Pro B70s, at the submitter's request. - `--max-context 65536` because at 200,000 not every expert fits on a plain two-card layer split of upstream main. - The kernel pays only when neighbouring selections overlap. One chunk of 8,192 positions at 24K-32K context: the FP32 kernel 137-152 ms, the union kernel 38.7 / 82.3 / 146.6 ms when the union of 8 selections is 1.15x / 2.67x / 5.22x one position's cells (docs/INTEL.md measured 3.3x on an 80K prompt). With independent selections it is no faster (512 positions at 32K, union 5.3x: 12.4 vs 11.0 ms). - About 80 MB of device memory per card that reads a prompt, allocated at the first prompt (it comes out of the `--vram-reserve-mib` headroom). - A cell is stored as an int32 with its 8-position mask in a separate byte, so any `--max-context` the engine accepts works. - Not included: an earlier version of the kernel (S = Q K^T, 32-cell chunks; the same error, 15-20% slower per chunk) and a knob for the groups per launch. - Not tested: the JIT build, other Xe2 parts (refused by the device check), `--kv-resident` (non-resident pages are masked as in the FP32 kernel; the parity test covers that case). 🤖 Generated with [Claude Code](https://claude.com/claude-code)
Sur le site
Liens install, modèles, releases.