Pull requests / #2437

#2437 SYCL: the IQ4_XS expert dot reads its codebook from registers

open · @deriknel · 0 comments · View on GitHub

BenchmarksMulti-GPUNVIDIA / CUDAModels & quantsWindows

Description

## Title
Issue: none filed.

## Summary
`vec_dot_iq4_xs_q8_1`, the one-entry dot the expert gate/up lane kernels use for IQ4_XS, still looks each 4-bit value up with ggml's `get_int_from_table_16`, which reads `kvalues_iq4nl` from global memory on every lookup. `vec_dot_iq4_nl_q8_1` and IQ4_XS's split kernel (`Split<23>`) already get the same ints from `iq4nl_pair`, which keeps the codebook in registers. This change makes the IQ4_XS dot do the same. On two Arc Pro B70s the IQ4_XS gate/up kernel goes from 378 to 171 µs per launch, with the same output.

## What changed
`sycl/src/kernels/cuda/iq_kernels.dp.cpp`, `vec_dot_iq4_xs_q8_1`: `iq4nl_pair(aux_q4)` in place of `get_int_from_table_16(aux_q4, kvalues_iq4nl)`, under the existing compile-time `STRATA_IQ4NL_FAST` switch (on by default). With `-DSTRATA_IQ4NL_FAST=0` the old lookup stays. Both return the same ints, so the output does not change.

## Extra Notes
Measured on 0.1.44.1 (296b59d2) with #2433 and #2436: 2x Arc Pro B70, Swift 1.5 Flash-Next IQ3_S, layer split 26, every expert in VRAM, `--pipeline-windows 2`. This quant's expert gate/up rows are IQ2_S, IQ3_XXS and IQ3_S, and IQ4_XS on one layer of the 48.

- unitrace over the same requests, `launch_gu_port<T, 8, true>`, average per launch:

  | gate/up format | launches | before | after |
  |---|---|---|---|
  | IQ2_S (22) | 12,660 | 94.9 µs | 95.6 µs |
  | IQ3_XXS (18) | 10,761 | 104.9 µs | 105.5 µs |
  | IQ3_S (21) | 6,330 | 107.9 µs | 108.8 µs |
  | IQ4_XS (23) | 633 | 378.0 µs | 171.4 µs |

  The "after" build also carried a change to the hyper-connection kernels, which do not touch these.
- Decode, `STRATA_DECODE_TIMING`, a 400-token code answer: 27.21 → 26.99 ms per window pipelined, 34.3 → 34.0 serial. Bench 79.9 / 114.4 / 89.5 → 80.4 / 115.4 / 90.0 tok/s (prose / code / a long answer). The gain is small here because only one layer is IQ4_XS; a quant with more IQ4_XS expert layers gains more.
- `iq_parity` with fixtures: 0 failures on both cards. `native_grouped_parity`: 0 failures. Greedy text on 5 prompts identical to the build without the change.
- Built and tested the SYCL target only; the change is in `sycl/`.

We investigated (myself and Claude Opus 5.5 in Claude Code) and I reviewed the findings and ran every test on my two Arc Pro B70s on my CachyOS machine.

Related on this site

Quick links to related pages.