Pull requests / #1824
#1824 cpu: Q2_0 rows on AVX-only CPUs (q2_avx1.cpp) instead of ggml-cpu's scalar dot
open · @xrip · 0 commentaires · Sur GitHub
Description
## What The older-CPU build (`STRATA_ISA_FLOOR=avx`) on an AVX-only CPU (Sandy / Ivy Bridge, Xeon E5 v1/v2): a native pack's Q2_0 expert rows run on a new AVX kernel (`src/kernels/cpu/q2_avx1.cpp`) instead of ggml-cpu's Q2_0 `vec_dot`, which is scalar on x86. Bitwise the same results as the scalar reference. AVX2 / AVX-512 CPUs and CPUs without AVX run the same code as before. ## Why In the IQ3_XXS pack, Q2_0 is the down projection of 30 of the 48 layers, so on these CPUs most CPU expert work went through a scalar loop. ## Change - `q2_native_kernels()` also accepts an AVX-only CPU (`cpu_avx1_ok()`). - `q2_rows_any`: AVX-512 -> AVX2 -> **AVX1 (new)** -> scalar. `act_quant_any` uses a scalar Q8_1 quantizer without AVX2. - `q2_0_gguf_rows_multi_avx1`: compiled with `-mavx` / `/arch:AVX` only (no FMA). The 2-bit codes are unpacked once per block for all tokens; SSSE3 `maddubs` / `madd` integer dots (exact: a pair sums to at most 762, no saturation); an exact half -> float; the float steps in the scalar order, so the bits equal `q2_0_gguf_rows_multi_scalar`. - The router lookahead keeps 0.1.41's `kq_avx1.cpp`. - New test `q2_avx1_parity` (ctest; exit 77 without AVX). ## Measured 2x Xeon E5-2670 v2 (Ivy Bridge-EP), GCC 14.2: - `q2_avx1_parity`: bitwise equal to the scalar reference for all 65,536 half scales and 216 random cases (1-12 tokens, partial row ranges, padded and unaligned rows, edge and zero-scale activations). - One expert's down rows (2560 x 640) on one core: | tokens | scalar | AVX1 | |---:|---:|---:| | 1 | 3.86 ms | 0.41 ms (9.4x) | | 4 | 13.2 ms | 0.78 ms (16.9x) | Measured on a fork based on 0.1.38. On current `main` (0.1.41) it was checked on a Ryzen 7 5700X with `STRATA_FORCE_ISA=avx`: `q2_avx1_parity` passes (12.5x over scalar at 1 token), and so do all CPU tests. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
Sur le site
Liens install, modèles, releases.