Pull requests / #1711
#1711 sycl: opt-in grouped XMX expert GEMM for the prompt path (STRATA_PF_XMX), with a parity test
open · @aslater3 · 0 comentários · No GitHub
AMD / HIPNVIDIA / CUDAModels & quantsLinux
Descrição
Adds an opt-in grouped XMX GEMM for the prompt path's routed experts on Intel Arc. With `STRATA_PF_XMX` unset, the default path does not change. ## What it changes - `STRATA_PF_XMX=1`: dequantizes a group of up to 16 routed experts to FP16, then runs one `joint_matrix` launch per product. A standalone measurement against oneMKL on uneven routing: down 1.75–6x faster, gate/up 0.8–3x. Above `STRATA_PF_XMX_GU_MAX=250` tokens per expert, gate/up stays on oneMKL. - `STRATA_PF_XMX=2`: the same GEMM, but the kernel decodes the IQ3_XXS / IQ3_S / IQ2_S / IQ4_XS gate/up rows and the IQ4_NL / Q2_0 down rows itself, so there is no FP16 staging pass. - New `xmx_group_parity` test (built with `STRATA_SYCL_PARITY=ON`): checks every supported type against dequant+oneMKL. The second commit fixes the in-kernel decode after rebasing onto current `main`. `main` now has the IQ2/IQ3/IQ4_NL/IQ1_M decoders write values `[8·tid, 8·tid+8)` in coalesced runs. The grouped kernel was still passing the old `sub*8 + o32/32` thread index, so four of the six types decoded garbage (relative error ≈10). It now passes `tid = o >> 3` for every type except IQ4_XS, which keeps its own layout. ## Measured On an Arc A770 16 GB (xe driver, oneAPI 2025.3, Linux 7.0), `xmx_group_parity` on this branch: ``` type 18 gate/up rows 1365: dequant+oneMKL 2.593 ms | fused xmx 1.800 ms (1.44x) relerr 9.9e-07 OK type 21 gate/up rows 1365: dequant+oneMKL 1.507 ms | fused xmx 1.765 ms (0.85x) relerr 1.1e-06 OK type 22 gate/up rows 1365: dequant+oneMKL 1.238 ms | fused xmx 1.330 ms (0.93x) relerr 1.5e-06 OK type 23 gate/up rows 1365: dequant+oneMKL 2.498 ms | fused xmx 1.516 ms (1.65x) relerr 1.8e-06 OK type 20 down rows 1365: dequant+oneMKL 0.825 ms | fused xmx 0.773 ms (1.07x) relerr 4.6e-07 OK type 42 down rows 1365: dequant+oneMKL 0.700 ms | fused xmx 0.603 ms (1.16x) relerr 1.5e-07 OK all OK ``` Without the second commit, types 18, 21, 22 and 20 fail with relative error ≈10. End to end, on the Coder IQ1_M model with an 8K prompt: - Earlier local measurements, before the rebase: 0.5–3K-token prompts were 14–29% faster with `STRATA_PF_XMX=2`, and 8K/32K prompts 2–5% faster. These were not repeated on this branch. - This branch, `main` + #1602 on the A770: a same-day 8K run did not show a gain. So it stays opt-in. ## Not tested - B-series (Battlemage) and other Alchemist cards. - The CUDA and HIP builds: these files are SYCL only (`sycl/`), plus one declaration in `include/strata/kernels/iq_kernels.hpp` and a new `include/strata/prefill/xmx_moe.hpp`, neither of which the CUDA/HIP sources include. - A full logit comparison end to end with `STRATA_PF_XMX=2`. The parity test covers the kernel only.
No site
Links install, modelos, releases.