Pull requests / #2340

#2340 sycl mmvq: Q6_K decode kernels no longer ALU-bound; Q6_K output heads as aligned planes; int4 activation loads

open · @leadgtr7 · 0 comentarios · En GitHub

BenchmarksAMD / HIPNVIDIA / CUDAModels & quantsWindows

Descripción

**What it changes (SYCL port only, `sycl/`):**
- **Q6_K six-bit offset as SWAR.** The Q6_K wide kernel (and the planar one below) computed `v - 32` with dpct's per-byte saturating `char4` subtract (`dpct::vectorized_binary(sub_sat)`), which compiles to a per-byte loop on Xe. No byte can saturate there, so a 4-operation SWAR form gives the same bits. Q6_K dense matvec: 59.5 -> 30.8 us (1 row), 77.4 -> 45.5 us (4 rows).
- **`STRATA_Q6_PLANAR` (on by default, `=0` turns it off).** The Q6_K LM head and the MTP draft head are converted at load into ql / qh / scales / d planes (the same bytes; the GGUF-layout allocation is freed, so VRAM does not change), so each lane's loads are single aligned 16-byte loads. The draft head's subset is gathered from the planar head. `native_q6_k_mmvq` serves every registered matrix at any column count.
- **`ld_i4_a4`:** the Q8_1 activation quants (4-byte aligned) are read as one dword-vector load in the Q4_K / Q5_K / IQ4_XS wide kernels (Q5_K, 4 rows: 81 -> 68 us).

These are three parts of one kernel pass and are sent together. Each gives bitwise-equal outputs; I can split them if you prefer.

**What it leaves alone:** CUDA and HIP (`src/`), the prompt path, every other quant type's math. Greedy output is byte-identical to the build without the change (below).

**How it was checked**, on one Arc Pro B65 32 GB (Xe2, AOT `bmg-g31`, xe driver, oneAPI 2026.1 `strata-sycl-dev` image), Swift 1.5 GSQ-RCO IQ3_S with `--stream-experts` (about 13,000 of 24,576 expert slots in VRAM, the rest in the pinned host mirror), `--spec 4` + MTP, single slot, v0.1.44 (`ed4773eb`) with and without this commit, same config, 3 interleaved pairs, 3 greedy 256-token prompts + one ~9.4K-token prompt per arm:

| | v0.1.44 | + this PR |
|---|---|---|
| decode tok/s (per pair) | 37.0 / 36.8 / 36.8 | 39.3 / 39.5 / 39.0 |
| decode, mean | 36.9 | 39.3 (+6.6%) |
| prompt reading tok/s | 818 | 817 |
| greedy replies byte-identical | | 12 / 12 |

The kernel timings above were measured on 0.1.41 with the same kernels. The same change has been in our daily server since 0.1.41.

**Not tested:** Arc A-series (i915) and Arc B580/B570 (`bmg-g21`); models whose expert tensors are Q6_K (the Unsloth UD files); the SYCL kernel parity tests (not registered in our build image: no numpy). Windows (no SYCL launcher there).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

En el sitio

Enlaces a install, modelos, releases.