Issues / #2803

#2803 [Feature]: SYCL: faster small-batch decode GEMVs for Q4_K, Q5_K, Q6_K and IQ4_XS on the Arc Pro B70 (opt-in)

open · @hotschmoe · 0 Kommentare · Auf GitHub

BenchmarksMulti-GPUModels & quantsDocumentation

Beschreibung

### What do you want to do?

Decode faster on an Intel Arc Pro B70 with the SYCL port by making the dense GEMVs of a decode window
(`native_mmvq` at 1-8 columns: the projections, the shared expert, the verify window and the 248,320-row output
head) read their weights closer to the card's memory bandwidth.

What I see today (one B70, `native_mmvq` called directly with random weights, distinct weight copies per graph so
L2 never holds a matrix):

- On the large matrices (2560 x 6144 / 10240 / 12288, 6144 x 2560, the head) the Q4_K, Q5_K and Q6_K kernels stream
  31-72% of the card's 608 GB/s at 1-2 columns and 18-41% at 6 columns; IQ4_XS 41-54% and 30-34%.
- The Q5_K head (2560 x 248,320) takes 1,063 us at 1 column and 2,691 us at 6.
- The time is not memory-bound: with the matrix held in L2, Q4_K 2560 x 10240 at 6 columns still took 82 us against
  94 us from DRAM. The wide kernels re-read every column's activations for every row (at 6 columns about 10x the
  weight bytes in L1 loads, issued one at a time), so they are bound by instruction issue at 4-6 columns.

What I'd like: an opt-in SYCL kernel family for these four formats that loads each column's activations once per
work-group and streams rows against them, with the default path left exactly as it is. A version of this on two
B70s (upstream main, `--layer-split 30`, Qwen3.8-Flash-Next ISTA GSQ-RCO IQ3_XXS, `--spec 4 --mtp`, one slot)
decodes 79.5 -> 95.0 tok/s (median of 3), with the K-quant matrices at 74-97% of bandwidth at 1-2 columns and the
head at 759 us (1 column) and 1,175 us (6 columns). It is not bitwise equal to the default kernels (another FP32
summation order), but a multi-column call stays bitwise equal to one-column calls.

Workaround today: none. The existing switches do not move decode: `STRATA_MMVQ_WIDE_K_RPW=2` measured 54.31 ->
54.00 tok/s (commit 2a0d2f55), and the SIMD16 builds (`STRATA_MMVQ_SG`) gave no gain beyond noise (docs/INTEL.md).

Prepared with AI assistance (Claude Code): the code, the measurements and this text were produced by Claude on the submitter's two Arc Pro B70s, at the submitter's request.

Mehr auf der Site

Links zu Install, Modellen, Releases.