#2802 sycl: cheaper IQ decodes in the grouped expert kernels (STRATA_IQ_DECODE_FAST=1, opt-in, bitwise)

本文

## Title
sycl: cheaper IQ decodes in the grouped expert kernels (STRATA_IQ_DECODE_FAST=1, opt-in, bitwise)

Issue: Resolves #2801

## Summary

The SYCL lane kernels that read the routed experts (`native_gu_kernel` / `native_down_kernel`, 8 lanes a row) are
ALU-bound on an Arc Pro B70: 280-480 instructions per 32-weight part for the part's 8 dp4a, reading the experts at
19-44% of DRAM bandwidth. Most of it is the dpct per-byte emulation of `byte_level_permute` in the Q2_0 dot (476 per
part, 157 of them `mov`), the compare/select chain of the IQ4_NL codebook lookup, the popcount/multiply ksigns
unpacking, and 64-bit `kbx * sizeof(block)` address products every iteration.

`STRATA_IQ_DECODE_FAST=1` switches the seven IQ expert formats (IQ2_XXS, IQ2_XS, IQ3_XXS, IQ3_S, IQ2_S gate/up;
IQ4_NL, Q2_0 down) to kernels that produce the same bits with fewer instructions. Off by default: without the flag
the engine runs exactly the code it ran before (the hook is the first statement of `launch_gu_port` /
`launch_down_port` and returns at once). SYCL only.

Measured on two Arc Pro B70 (32 GB, kernel 7.1, Compute Runtime 26.22.38646.4, oneAPI 2026.1, AOT `bmg-g31`),
upstream main 296b59d2 against this branch, Qwen3.8-Flash-Next GSQ-RCO IQ3_XXS, `--layer-split 30 --prefill 8192
--max-context 65536 --kv int8 --spec 4 --mtp <BF16 draft layer> --mtp-max-t 4 --spec-min-p 0 --stream-experts
--expert-cache auto`, one slot, greedy, a fixed prompt with a 512-token coding reply:

- decode: main 81.8 / 79.5 / 78.7 tok/s (3 runs, median 79.5); this branch, flag unset, 81.7 / 79.3; with
  `STRATA_IQ_DECODE_FAST=1` 88.1 / 85.6 / 84.7 (median 85.6, +7.7%);
- the output tokens and the draft acceptance counts are identical in every run, flag on or off, so the gain is
  kernel time alone; a turn at a 40,796-token context: 76.0 -> 81.4 tok/s (+7.1%), identical output;
- prompt reading unchanged (4K 1189 -> 1192 tok/s, 32K 1956-2004 -> 1964-2009): the change is decode-only;
- `native_grouped_parity --iq`: 0 failures (15 cases bitwise equal); plain `native_grouped_parity`: 0 failures;
  per-card health after every run passed. Two concurrent streams were not tested (main does not take `--batch-mtp`
  on a layer split).

On a development tree, a grouped-call microbenchmark on one B70 (48 layers per graph streaming their own experts,
us per call, new against current, bitwise equal in every row) put the gain per layer type at: IQ2_XXS/Q2_0
114.8 -> 59.8 to 248.7 -> 147.3, IQ3_S/Q2_0 105.1 -> 70.3 to 256.9 -> 175.1, IQ2_S/Q2_0 97.0 -> 67.2 to
237.3 -> 161.9, IQ2_XS/IQ4_NL 135.6 -> 119.6 to 242.6 -> 205.4, IQ3_XXS/IQ4_NL 146.6 -> 127.9 to 253.4 -> 221.3
(the shapes of a two-card serve at 1 to 4 streams); the Q2_0 down product alone 63.7 -> 26.4 us.

## What changed

- `sycl/src/kernels/cuda/iq_kernels.dp.cpp` (+426 lines, inside the anonymous namespace, before the lane launchers):
  - `IqFast<TY>` decode traits for types 16, 17, 18, 21, 22, 42, 20. Each `prep` yields the same integers as
    `Multi<TY>::prep` / the `vec_dot_*_q8_1` dot and reuses the same `acts` and `finish` (IQ2_XXS and IQ2_XS, which
    have no `Multi<>`, get a `finish` with the statements of their `vec_dot` in the same order). Inside `prep`: the
    IQ2_XXS / IQ2_XS ksigns masks from a 2 KB table (`kIqKsignMask` = `unpack_ksigns` and the two `swar_ne4`); a
    signed grid byte as `(g ^ m) + (m & 0x01010101)`, which equals `swar_sub4(g ^ m, m)` for grid bytes under 128
    (every byte of the five IQ grids in ggml-common.h is 1..62); Q2_0 fields spread by shifts and masks and biased
    with `(t + 0x7F7F7F7F) ^ 0x80808080`; the IQ4_NL codebook through four 256-entry position tables (`kIqNlPos`,
    8 KB); IQ3_XXS / IQ3_S / IQ2_S keep their ALU sign masks (a table measured slower for them) and take aligned
    8-byte loads.
  - `rows_dot_fast<TY, LANES, R, E>`: R rows x E entries per lane, walking block pointers (with `LANES % ipb == 0` a
    lane's part index is fixed), the k order, per-entry float sum order and `lanes_sum` of `row_dot_lanes` /
    `row_dot_multi_iq`.
  - `native_gu_kernel_fast` / `native_down_kernel_fast`: the lane kernels with those dots, two rows per lane in
    16-lane sub-groups (the one-row SIMD32 form spilled registers and measured slower), entries in chunks of 4, 2 and
    1 as `row_entries`; IQ2_XXS thereby gets a multi-entry form (the current kernel decodes it once per entry).
  - `iq_fast_on(type)`: `STRATA_IQ_DECODE_FAST` read once at the first expert call (one stderr line when on);
    `STRATA_IQ_DECODE_FAST_FMTS=<ggml ids>` restricts it to some formats for an A/B. `launch_gu_fast` /
    `launch_down_fast` launch the kernels at 8 lanes a row only (other `STRATA_GU_LANES` / `STRATA_DOWN_LANES` keep
    the current kernels).
  - One hook line at the top of `launch_gu_port` and `launch_down_port`; `native_expert_set_iq_decode_fast(int)`
    for the parity test (declared there, like `native_expert_set_kq_split`; no header change).
- `sycl/src/kernels/cuda/iq_decode_tables.inc` (new): the two tables, with the formulas that generate them.
- `sycl/src/kernels/native_grouped_parity.cpp`: `--iq` mode: fast against current, bitwise, every IQ gate/up format
  with both IQ down formats at 512 x 256 and the model's five expert pairs at 2560 x 640, groups of 1..6 entries
  (the chunks of 4 and 2 and the single tail), entries from 0 and from 7, scattered destinations, stale scratch, the
  rows a call must not write included.
- `sycl/CMakeLists.txt`: ctest `native_grouped_parity_iq_decode`.
- `docs/INTEL.md`: a section on the flag with the measurements above.

Left alone: `include/`, `src/`, the CUDA and HIP backends, every kernel the default path runs, the K-quant / Q5_1 /
Q8_0 expert formats, the MMVQ kernels, the prompt path. Not included: a table form of the IQ3 sign masks and a
bit-slice IQ4_NL select (both measured slower), the one-row SIMD32 variant, and gate/up Q2_0 (not measured).

## Extra Notes

- Why the bits cannot move: each part's dp4a inputs are the same integers as before and the `finish` is the same
  expression in the same order; the per-entry float sums run in the same k order with the same `lanes_sum`. The
  parity test checks it on random blobs, the serve above end to end (identical tokens and acceptance counts).
- Built with the SYCL build (`sycl/CMakeLists.txt`): `strata` and `native_grouped_parity`. No CUDA or HIP file is
  touched.
- Prepared with AI assistance (Claude Code): the code, the measurements and this text were produced by Claude on the submitter's two Arc Pro B70s, at the submitter's request.

🤖 Generated with [Claude Code](https://claude.com/claude-code)