Issues / #2803
#2803 [Feature]: SYCL: faster small-batch decode GEMVs for Q4_K, Q5_K, Q6_K and IQ4_XS on the Arc Pro B70 (opt-in)
open · @hotschmoe · 0 comentarios · En GitHub
BenchmarksMulti-GPUModels & quantsDocumentation
Descripción
### What do you want to do? Decode faster on an Intel Arc Pro B70 with the SYCL port by making the dense GEMVs of a decode window (`native_mmvq` at 1-8 columns: the projections, the shared expert, the verify window and the 248,320-row output head) read their weights closer to the card's memory bandwidth. What I see today (one B70, `native_mmvq` called directly with random weights, distinct weight copies per graph so L2 never holds a matrix): - On the large matrices (2560 x 6144 / 10240 / 12288, 6144 x 2560, the head) the Q4_K, Q5_K and Q6_K kernels stream 31-72% of the card's 608 GB/s at 1-2 columns and 18-41% at 6 columns; IQ4_XS 41-54% and 30-34%. - The Q5_K head (2560 x 248,320) takes 1,063 us at 1 column and 2,691 us at 6. - The time is not memory-bound: with the matrix held in L2, Q4_K 2560 x 10240 at 6 columns still took 82 us against 94 us from DRAM. The wide kernels re-read every column's activations for every row (at 6 columns about 10x the weight bytes in L1 loads, issued one at a time), so they are bound by instruction issue at 4-6 columns. What I'd like: an opt-in SYCL kernel family for these four formats that loads each column's activations once per work-group and streams rows against them, with the default path left exactly as it is. A version of this on two B70s (upstream main, `--layer-split 30`, Qwen3.8-Flash-Next ISTA GSQ-RCO IQ3_XXS, `--spec 4 --mtp`, one slot) decodes 79.5 -> 95.0 tok/s (median of 3), with the K-quant matrices at 74-97% of bandwidth at 1-2 columns and the head at 759 us (1 column) and 1,175 us (6 columns). It is not bitwise equal to the default kernels (another FP32 summation order), but a multi-column call stays bitwise equal to one-column calls. Workaround today: none. The existing switches do not move decode: `STRATA_MMVQ_WIDE_K_RPW=2` measured 54.31 -> 54.00 tok/s (commit 2a0d2f55), and the SIMD16 builds (`STRATA_MMVQ_SG`) gave no gain beyond noise (docs/INTEL.md). Prepared with AI assistance (Claude Code): the code, the measurements and this text were produced by Claude on the submitter's two Arc Pro B70s, at the submitter's request.
En el sitio
Enlaces a install, modelos, releases.