Pull requests / #2437
#2437 SYCL: the IQ4_XS expert dot reads its codebook from registers
open · @deriknel · 0 コメント · GitHub で見る
BenchmarksMulti-GPUNVIDIA / CUDAModels & quantsWindows
本文
## Title Issue: none filed. ## Summary `vec_dot_iq4_xs_q8_1`, the one-entry dot the expert gate/up lane kernels use for IQ4_XS, still looks each 4-bit value up with ggml's `get_int_from_table_16`, which reads `kvalues_iq4nl` from global memory on every lookup. `vec_dot_iq4_nl_q8_1` and IQ4_XS's split kernel (`Split<23>`) already get the same ints from `iq4nl_pair`, which keeps the codebook in registers. This change makes the IQ4_XS dot do the same. On two Arc Pro B70s the IQ4_XS gate/up kernel goes from 378 to 171 µs per launch, with the same output. ## What changed `sycl/src/kernels/cuda/iq_kernels.dp.cpp`, `vec_dot_iq4_xs_q8_1`: `iq4nl_pair(aux_q4)` in place of `get_int_from_table_16(aux_q4, kvalues_iq4nl)`, under the existing compile-time `STRATA_IQ4NL_FAST` switch (on by default). With `-DSTRATA_IQ4NL_FAST=0` the old lookup stays. Both return the same ints, so the output does not change. ## Extra Notes Measured on 0.1.44.1 (296b59d2) with #2433 and #2436: 2x Arc Pro B70, Swift 1.5 Flash-Next IQ3_S, layer split 26, every expert in VRAM, `--pipeline-windows 2`. This quant's expert gate/up rows are IQ2_S, IQ3_XXS and IQ3_S, and IQ4_XS on one layer of the 48. - unitrace over the same requests, `launch_gu_port<T, 8, true>`, average per launch: | gate/up format | launches | before | after | |---|---|---|---| | IQ2_S (22) | 12,660 | 94.9 µs | 95.6 µs | | IQ3_XXS (18) | 10,761 | 104.9 µs | 105.5 µs | | IQ3_S (21) | 6,330 | 107.9 µs | 108.8 µs | | IQ4_XS (23) | 633 | 378.0 µs | 171.4 µs | The "after" build also carried a change to the hyper-connection kernels, which do not touch these. - Decode, `STRATA_DECODE_TIMING`, a 400-token code answer: 27.21 → 26.99 ms per window pipelined, 34.3 → 34.0 serial. Bench 79.9 / 114.4 / 89.5 → 80.4 / 115.4 / 90.0 tok/s (prose / code / a long answer). The gain is small here because only one layer is IQ4_XS; a quant with more IQ4_XS expert layers gains more. - `iq_parity` with fixtures: 0 failures on both cards. `native_grouped_parity`: 0 failures. Greedy text on 5 prompts identical to the build without the change. - Built and tested the SYCL target only; the change is in `sycl/`. We investigated (myself and Claude Opus 5.5 in Claude Code) and I reviewed the findings and ran every test on my two Arc Pro B70s on my CachyOS machine.
関連リンク
インストール・モデル・リリースへの站内リンク。