#6262 [Engine] IQ4_NL down rows in 512-bit lanes, opt-in, same bits as today's paths

open · @superbaenawa · 0 Kommentare · Auf GitHub

BenchmarksSetup & installAMD / HIPNVIDIA / CUDAModels & quantsWindowsLinux

Beschreibung

## Problem and result

In Qwen3.8-Flash-Next GSQ-RCO IQ3_S, 39 of the 48 layers have IQ4_NL down projections. On an AVX-512 CPU, the pool runs their rows on `iq4nl256_down_rows` from two tokens on. Below the #152 rule, one token runs on ggml-cpu's AVX2 dot.

In 55dc17697 a 512-bit version of these rows was tried, and 6368dc68 dropped it 17 minutes later, because it measured 5-11% slower than `iq4nl256_down_rows` on Zen 4 at 2-4 tokens. I think that call was right. I rebuilt that kernel exactly as committed and measured it on my Zen 4: it is 13-15% slower than `iq4nl256_down_rows` at 2-4 tokens. The comment above `q50_512_down_rows` records this.

This PR tries a different shape. It does not use the 512-bit width to do the same work per block. It cuts the work per block (details under Approach). With `STRATA_IQ4NL512=1` (opt-in, default off), IQ4_NL down rows take these new rows. Each call gives the same bits as the path it replaces.

On a Ryzen 7 8700G (Zen 4), `pool_rows_bench` at the model's expert shape shows down rows taking 30.8 µs per expert instead of 72.6 at one token (2.36x), and 50.5 µs instead of 97.2 at two tokens (1.92x). The gain at 3, 4 and 8 tokens is 1.88x to 2.16x.

What that buys depends on memory bandwidth. These rows stream their weights from DRAM, and one thread of today's rows reads them at 12.7 GB/s (one token). On this machine (DDR5, 2 channels, about 66 GB/s), 5-6 threads of today's rows already fill the bus at one token, so the new rows cannot help there. With fewer threads than that, or more tokens per call, they read the same weights 1.9-2.4x faster until they reach the same limit (table under Validation). A machine with more bandwidth per pool thread (more channels, faster memory) stays below its limit for longer, and keeps more of the gain. On a desktop like this one, the new rows reach the bus limit with fewer threads: 2-3 instead of 5-6 at one token. In the engine with 3 pool workers, the down rows took 27% less thread time with the switch set. On this machine's own engine decode, the gain is within noise (+0.5%), because the pool is at the DRAM limit there with or without this change (details under Validation).

## Related discussion

- 55dc17697 / 6368dc68 (above), and the comment above `q50_512_down_rows`. This PR adds one clause to that comment and keeps its sentence.
- #4763 (open) has IQ4_NL rows in 512-bit lanes in ggml's order, and #5243 (draft) has a version too. I measured the #4763 kernel and the 55dc17697 kernel as committed, next to this one (numbers under Validation). I did not measure the #5243 version. I am not proposing to replace or close either one. If the maintainers prefer one of them, this PR could stay as measurements for that thread.
- I did not open a scope issue first, because this is opt-in and changes no default. If you would rather have it on by default, as `q50_512` is, that is a one-line change, but that decision is yours.

## Approach

Zen 4 executes a zmm instruction as two 256-bit halves. A 512-bit kernel only gains if it issues fewer 256-bit operations per block than the AVX2 kernel does. The 55dc17697 shape issued more: per block it kept the `abs` and the sign mask, and decoded each block with two 128-bit `vpshufb` and an insert.

The new rows do this instead (the integer lanes are the same integers as ggml's):
- **No abs or sign per block.** The codebook is looked up as w + 128 (unsigned), so `vpdpbusd` can take it directly. Each lane starts from -128 times the sum of that lane's four y values. That start value is computed once per call for each token, not once per block.
- **One lookup per block pair.** One `vpermb` (`vpermi2b` for two rows) places a block pair's nibbles, and one more `vpermb` looks up the codebook.
- **Wide loads with no mask.** Blocks are read four at a time (72 bytes) with two unmasked 64-byte loads that stay inside the four blocks.
- **Scales several at a time.** Four scales (eight in the one-token form) are converted and multiplied by d_y together.
- **Outside the block loop:** the several-token form handles up to 8 tokens in one pass over the weights, and sums two rows per register at the end. The one-token form walks 8 blocks a step and has its own instance for n = 640 (this model's expert width), so that loop has no tail.

Hot-loop counts from the same compiler (GCC 15.2, `-O3`), per block and row. "256-bit slots" counts a zmm instruction twice:

| rows (2 tokens) | vector instructions | 256-bit slots | measured, in cache |
|---|---|---|---|
| `iq4nl256_down_rows` | 30 | 30 | 1.00 |
| 55dc17697 `qd_rows<20, 2>` | 29 | 35 | 0.87x (slower) |
| #4763 `iq4nl512_rows<2>` | 21.5 | 30 | 0.96x (slower) |
| this PR, several tokens | 7.9 | 15.25 | **1.93x** |

#4763 has as few 256-bit slots as `iq4nl256_down_rows` but still measured 4% slower here. In its 2-token loop, 28 of the 54 instructions for a block pair handle the scales: each f16 scale is moved from a general register and converted on its own (`vmovd`, `vcvtph2ps` on one value, `vmulss`, a broadcast and an insert), for every block and token. I have not profiled it further, but converting several scales per instruction, as this PR does, might close most of that gap.

The accumulation order is unchanged, so the bits are unchanged:
- **Several tokens (`iq4nl256_down_rows`' order):** two rows per register (lanes 0-7 and 8-15), one 8-lane accumulator per row over its blocks in order, then that file's `hsum8`.
- **One token, and the window-invariant rows (ggml-cpu's dot):** block 2k in lanes 0-7, block 2k+1 in lanes 8-15, `fma(d_y * d_x, lane)`, then `hsum_float_8(accum1 + accum2)`. `iq4nl_reference_rows` gives the same bits.

How it is wired:
- **Switch and gates** (`native_expert.cpp`): the rows run only with `STRATA_IQ4NL512=1`, on `cpu_avx512_ok() && cpu_vnni_ok()` (F, BW, VL, DQ, VNNI, VBMI), for IQ4_NL down with no NVFP4 tail. Where `iq4nl256_down_rows` runs today, they replace it. Everywhere else, they replace ggml's dot only when `flavour_avx2()`. The one-token form returns false for an odd block count or n > 4096, and those calls keep today's path.
- **First-use check** (`iq4nl512_ok`, the same pattern as `q50_512_ok`): synthetic blocks at n = 64, 96, 160, 320, 448, 640 and 1088, 17 rows, 8 tokens (5 where n is not a multiple of 64). Together these reach every branch of both forms: the one-token form's 8-block steps, its 4-block step and its masked last pair, and the several-token form's 1-, 2- and 3-block masked tails. The standalone test below covers every block count from 1 to 130. The one-token form is compared with this build's dot, and the several-token form with `iq4nl256_down_rows`. On a mismatch it prints one line and keeps today's paths.
- **Scratch:** the several-token form keeps its activation copy on the stack up to n = 1024 (32 KB), and in a `thread_local` buffer above that.
- **`tools/check_isa_guard.py`:** `iq4nl512_down_rows` is listed with the wide public kernels, like `q50_512_down_rows`.
- **Prefetch:** it uses the file's `prefetch_ahead` (`STRATA_IQ_PREFETCH`). On Zen 4 it needs it: with the prefetch off, the several-token form gets slower from DRAM.

## Validation

Revision: upstream `main` `58ee32277` + this change. Ryzen 7 8700G (Zen 4, 8 cores, DDR5 2 channels), Linux, GCC 15.2. Preset `cpu-check-native` (Release, Strata quant layer). "base" is an unmodified build of `58ee32277` in the same configuration.

**Passed:**
- `ctest` in `cpu-check-native`: 155/155 passed, with the switch unset and with `STRATA_IQ4NL512=1` (the same 7 tests are skipped in base).
- `python3 tools/check_isa_guard.py build/cpu/libstrata_kernels_cpu.a`: 894 functions, 177 hold wide-ISA instructions, 0 not allowed.
- `cpu_dispatch_dump --nt-max 9 --experts 4` (default pairs, 4 of them with IQ4_NL down): base, this build with the switch unset, and this build with `STRATA_IQ4NL512=1` print the same 1033 lines. The same holds under each of `STRATA_IQ_MT_MIN=1`, `STRATA_IQ_MT_MIN=3`, `STRATA_NO_IQ4NL=1`, `STRATA_IQ_PREFETCH=0` and `STRATA_NO_IQ512=1`. A breakpoint confirmed that the new rows run in the dump with the switch set, and not without it.
- Bit-exactness outside the engine: a standalone harness (no Strata, no ggml) checks the kernel against ggml-cpu's IQ4_NL dot (AVX2 path, copied) and `iq4nl256_down_rows` (copied): 0 differences in 552,636 rows at 1-8 tokens, with ±127 edge activations. A second test puts the weights between `PROT_NONE` guard pages, with 1-130 blocks, 1-9 rows, 1-9 tokens and unaligned row ranges. It finds 0 differences, no reads outside the row, and no writes outside the range, at `-O3` and `-O2`, with `STRATA_IQ_PREFETCH` at 0, 2048 and 4096, and under ASan + UBSan. A third test sets the f16 block scales to 0, -0, ±inf, subnormals and NaN: 0 differences. The tests do catch mistakes: a one-byte index error gives 22,780 differences, and a 4-byte overread hits the guard page.
- One thread, µs per expert for the down rows, 5 interleaved rounds per arm (each the median of 9 passes over 48 experts). Median (min-max):
  `taskset -c 2 pool_rows_bench --pair iq3_s/iq4_nl --embd 2560 --ff 640 --nt 1,2,3,4,8 --rounds 9`

  | tokens | base | switch unset | `STRATA_IQ4NL512=1` |
  |---|---|---|---|
  | 1 | 72.6 (72.3-72.7) | 72.5 (72.4-72.7) | **30.8** (30.7-31.0) |
  | 2 | 97.2 (97.1-97.3) | 97.3 (97.0-97.4) | **50.5** (50.4-50.7) |
  | 3 | 129.3 (129.2-129.6) | 129.5 (129.2-129.7) | **68.7** (68.5-68.7) |
  | 4 | 163.4 (163.2-163.5) | 163.6 (163.4-163.8) | **85.7** (85.6-85.9) |
  | 8 | 326.6 (326.3-326.9) | 326.7 (326.5-327.1) | **151.0** (150.9-151.2) |

  All 15 runs printed `all rows identical`. The gate/up times are the same in all three arms.
- Pool threads, weights streamed from DRAM (down rows, 4 GB of blobs), GB/s of weights, median of 5 interleaved rounds (largest spread between rounds 2.4%). base / `STRATA_IQ4NL512=1` (ratio):
  `kq_cpu_bench --pairs iq3_s/iq4_nl --threads 1,2,3,4,5,6,7,8,12,16 --nt 1,2,3,4,8 --part down --mb 4096 --reps 3`

  <details><summary>Table: 1-16 threads, 1-8 tokens</summary>

  | threads | 1 token | 2 tokens | 3 tokens | 4 tokens | 8 tokens |
  |---|---|---|---|---|---|
  | 1 | 12.7 / **29.8** (2.35) | 9.4 / **18.3** (1.95) | 7.1 / **13.5** (1.90) | 5.7 / **10.7** (1.88) | 2.8 / **6.1** (2.18) |
  | 2 | 24.8 / **57.6** (2.32) | 18.6 / **35.6** (1.91) | 14.1 / **26.3** (1.87) | 11.2 / **21.2** (1.89) | 5.6 / **12.1** (2.16) |
  | 3 | 36.8 / **66.3** (1.80) | 27.5 / **51.7** (1.88) | 20.8 / **38.9** (1.87) | 16.5 / **31.3** (1.90) | 8.3 / **17.9** (2.16) |
  | 4 | 48.6 / **67.1** (1.38) | 36.4 / **64.1** (1.76) | 27.5 / **51.2** (1.86) | 21.9 / **41.4** (1.89) | 11.0 / **23.6** (2.15) |
  | 5 | 60.1 / 66.8 (1.11) | 45.0 / **66.6** (1.48) | 34.1 / **63.1** (1.85) | 27.2 / **51.4** (1.89) | 13.7 / **29.4** (2.15) |
  | 6 | 66.3 / 66.7 (1.01) | 53.7 / **66.7** (1.24) | 40.6 / **66.4** (1.64) | 32.4 / **60.9** (1.88) | 16.2 / **35.0** (2.16) |
  | 7 | 66.6 / 66.5 (1.00) | 62.1 / 66.6 (1.07) | 47.2 / **66.5** (1.41) | 37.5 / **66.3** (1.77) | 18.8 / **40.5** (2.15) |
  | 8 | 66.5 / 66.3 (1.00) | 66.4 / 66.9 (1.01) | 53.6 / **66.6** (1.24) | 42.6 / **66.4** (1.56) | 21.4 / **46.1** (2.15) |
  | 12 | 65.9 / 65.9 (1.00) | 66.0 / 66.4 (1.01) | 53.4 / **66.5** (1.25) | 42.6 / **66.3** (1.56) | 21.3 / **38.6** (1.81) |
  | 16 | 65.5 / 65.7 (1.00) | 65.4 / 65.9 (1.01) | 53.5 / **66.2** (1.24) | 43.0 / **65.9** (1.53) | 21.4 / **31.5** (1.47) |

  </details>

  Below this machine's DRAM limit (about 66 GB/s), each thread keeps the one-thread ratio. Once a column reaches the limit, both arms read at the limit. Today's rows need 5-6 threads to fill this bus at one token and 7-8 at two. The new rows need 2-3 and 4-5, so the same bandwidth takes fewer cores (engine numbers with 3 workers below). On a machine with more channels, the same columns reach the limit later, if at all, so more of the left-hand ratios remain. I have not measured such a machine. With SMT (12 and 16 threads) at 8 tokens, 

Mehr auf der Site

Links zu Install, Modellen, Releases.