Pull requests / #2638

#2638 cpu: decline IQ4_NL down rows for the AVX1 kernels; two cleanups

open · @nashcap · 0 comentarios · En GitHub

BenchmarksAMD / HIPNVIDIA / CUDAModels & quantsSecurity

Descripción

On CPUs that have AVX but not AVX2, IQ4_NL down rows should be computed by ggml's own dot, not by our 128-bit kernel. ggml is faster there, and always has been. That is the point of this PR: one line in the dispatch. Two small cleanups follow.

## Why these commits exist

#1697 and #1701 landed in v0.1.42 (you ported them with my authorship). I kept working on the same files after that and pushed three more commits to the #1697 thread, and they were never ported. This PR is those three, rebased onto v0.1.44.1. The diff touches only `iq_avx1.cpp`, `q2_avx1.cpp` and `iq_avx1_parity.cpp`.

## 1. IQ4_NL down rows: decline them (the fix)

The other formats this file serves have no hand-written x86 dot in ggml-cpu, which is why the 128-bit kernel wins them. IQ4_NL is the exception: ggml-cpu has a hand-written `__AVX__` dot for it (`ggml_vec_dot_iq4_nl_q8_0`, two blocks per iteration, 256-bit float accumulation). That one beats our kernel on this CPU, and I could not close the gap - I matched its two-block shape and shared the decode across tokens, and it still came out behind.

So the dispatch stops choosing our kernel for that format:

```c
-bool iq128_down_supported(int type) noexcept { return type == 20 || type == 42; }
+bool iq128_down_supported(int type) noexcept { return type == 42; }
```

The kernel stays built and parity-tested; only the dispatch declines it.

Why it is worth a PR: on the IQ3_S pack, 24 of the 33 down layers are IQ4_NL. Today an AVX1 CPU runs the slower of the two kernels for all 24 of them.

Measured as `ms/expert` with `./iq_avx1_parity --bench --mb 96 --nt 1,4 --cpu 12`, two runs of one binary:

| down rows | who wins | margin |
|---|---|---|
| IQ4_NL (type 20) | **ggml** | 0.78-0.93x, every run, never above 1.0 |
| Q2_0 (type 42) | ours | 3.5-5.4x - ggml-cpu has no x86 dot for this type at all |

The bench now prints gate/up and down separately because of this finding. The combined line had been hiding a 0.78x down path inside a 2.1x result.

## 2. Micro-optimizations, all bit-exact

- fp16 to fp32 without F16C: for normal exponents the conversion is one integer shift and add, so that path is taken branchlessly. Zero, subnormal, inf and NaN still go through the full converter.
- Q2_0 down rows: the activation's per-lane `sum(q)` does not depend on the row, so it is computed once per call instead of once per row.
- one `cvtepi32` instead of two for the int32 pair sums.
- two rows through the block loop at a time. At NT=1 the decode chain (index byte, codebook lookup, `madd`) is about 20 cycles of pure latency with nothing to overlap, so the kernel runs latency-bound at roughly a quarter of its issue rate. A second row gives the scheduler something independent to do in the meantime.

(NT is the number of tokens in one verify window that route to the same expert, set by the router. Live decode is almost entirely NT=1.)

Net effect: gate/up rows measure 1.18-1.41x against ggml at NT=1, and 2.0-2.8x at NT=4.

## 3. The two-row form only at NT=1

That pairing pays only at NT=1. At NT >= 2 the row pair plus two token accumulators does not fit in 16 XMM registers and spills: neutral-to-noisy at NT=2, and going further is actively bad - R=3 measured a wash, R=4 measured -42.6% at NT=1 on IQ3_S. So the policy is R=2 at NT=1 and R=1 otherwise. `dot_rows` had no callers left once that path existed, so it is removed.

## Answers do not change

Every change here is either bit-exact by construction (the fp16 identity, the shared `sum(q)`, the fused conversion) or a dispatch decline. Nothing touches the pack format or the converter.

## Verification

Xeon E5-2470 v2 (AVX, no AVX2 / FMA / F16C / AVX-512), RTX 5060 Ti 16 GB, v0.1.44.1 source build with `-DSTRATA_ISA_FLOOR=avx`:

```
iq_avx1_parity: all rows match        (8 format pairs: gu 18/21/22/23 x down 20/42)
q8k_quant_parity: AVX1 copy identical
pool tasks: 168 bitwise comparisons passed
q2_bitplane_parity: PASS
check_isa_guard: 421 functions, 69 hold wide-ISA instructions, 0 not allowed
```

A live profile of the running engine shows the dispatch doing what the code says: `row_dot_r<18,1,2>` and friends at NT=1, `row_dot_r<18,2,1>` at NT>=2, and `ggml_vec_dot_iq4_nl_q8_0` carrying the IQ4_NL down rows at 7.2% of samples while `iq128_down_rows` sits at 2.6% - which is the 24/9 split.

One honest caveat about the numbers: single-format gate/up ratios at NT=1 are noisy (iq3_xxs measured 1.19x on one run and 0.90x on the next). The down-row numbers, which are the argument of this PR, were stable across runs.

En el sitio

Enlaces a install, modelos, releases.