Pull requests / #1824

#1824 cpu: Q2_0 rows on AVX-only CPUs (q2_avx1.cpp) instead of ggml-cpu's scalar dot

open · @xrip · 0 Kommentare · Auf GitHub

Multi-GPUModels & quants

Beschreibung

## What

The older-CPU build (`STRATA_ISA_FLOOR=avx`) on an AVX-only CPU (Sandy / Ivy Bridge, Xeon E5 v1/v2): a native pack's
Q2_0 expert rows run on a new AVX kernel (`src/kernels/cpu/q2_avx1.cpp`) instead of ggml-cpu's Q2_0 `vec_dot`, which
is scalar on x86. Bitwise the same results as the scalar reference. AVX2 / AVX-512 CPUs and CPUs without AVX run the
same code as before.

## Why

In the IQ3_XXS pack, Q2_0 is the down projection of 30 of the 48 layers, so on these CPUs most CPU expert work went
through a scalar loop.

## Change

- `q2_native_kernels()` also accepts an AVX-only CPU (`cpu_avx1_ok()`).
- `q2_rows_any`: AVX-512 -> AVX2 -> **AVX1 (new)** -> scalar. `act_quant_any` uses a scalar Q8_1 quantizer without AVX2.
- `q2_0_gguf_rows_multi_avx1`: compiled with `-mavx` / `/arch:AVX` only (no FMA). The 2-bit codes are unpacked once
  per block for all tokens; SSSE3 `maddubs` / `madd` integer dots (exact: a pair sums to at most 762, no saturation);
  an exact half -> float; the float steps in the scalar order, so the bits equal `q2_0_gguf_rows_multi_scalar`.
- The router lookahead keeps 0.1.41's `kq_avx1.cpp`.
- New test `q2_avx1_parity` (ctest; exit 77 without AVX).

## Measured

2x Xeon E5-2670 v2 (Ivy Bridge-EP), GCC 14.2:

- `q2_avx1_parity`: bitwise equal to the scalar reference for all 65,536 half scales and 216 random cases (1-12
  tokens, partial row ranges, padded and unaligned rows, edge and zero-scale activations).
- One expert's down rows (2560 x 640) on one core:

  | tokens | scalar | AVX1 |
  |---:|---:|---:|
  | 1 | 3.86 ms | 0.41 ms (9.4x) |
  | 4 | 13.2 ms | 0.78 ms (16.9x) |

Measured on a fork based on 0.1.38. On current `main` (0.1.41) it was checked on a Ryzen 7 5700X with
`STRATA_FORCE_ISA=avx`: `q2_avx1_parity` passes (12.5x over scalar at 1 token), and so do all CPU tests.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Mehr auf der Site

Links zu Install, Modellen, Releases.