Pull requests / #2215
#2215 cpu: one token's IQ3_XXS/IQ3_S/IQ2_S gate/up rows in ggml's exact bits, faster (same output; STRATA_CPU_SINGLE_EXACT=0 keeps ggml's dot)
open · @aslater3 · 0 commentaires · Sur GitHub
BenchmarksAMD / HIPNVIDIA / CUDAModels & quantsDocumentation
Description
## Title Issue: none (CPU decode speed on AVX2 machines without AVX-512) ## Summary On an AVX-2 CPU without AVX-512, a CPU expert that serves one token runs ggml-cpu's single-token `vec_dot` (#152). This adds a gate/up kernel for IQ3_XXS / IQ3_S / IQ2_S that uses the AVX-2 rows' decode but keeps **ggml's float order exactly** (the same per-lane integer sums, `d = x.d * y.d` and one fmadd per block, the format constant after `hsum_float_8`'s tree), so the output does not change, only the time: one-token gate/up -27% (IQ3_S) and -6% (IQ3_XXS) per expert on a Ryzen 7 3800X. `STRATA_CPU_SINGLE_EXACT=0` keeps ggml's dot. ## What changed - `src/kernels/cpu/iq_avx2.cpp`, `include/strata/kernels/cpu/iq_avx2.hpp`: `iq256_gu_single_exact` (+ `row_dot_single`, `gu_single`, `hsum8_ggml`). Upstream's multi-token kernels are untouched. - `src/kernels/cpu/native_expert.cpp`: `native_gu_rows` takes it for `nt == 1`, types 18/21/22, only where it would have reached ggml's dot (after the multi-token checks, so `STRATA_IQ_MT_MIN` / `STRATA_IQ3S_MT1` behave as before), the CPU has AVX2 and no AVX-512, **and** ggml-cpu itself was built for AVX2 and not AVX-512 (`ggml_cpu_has_*`: a `STRATA_ISA_FLOOR` build's ggml dots round differently). `STRATA_NO_IQ256` still means ggml's dot. Down rows are unchanged. - `src/kernels/cpu/iq_single_parity.cpp` (new): the kernel and `native_gu_rows` at one token against ggml's `vec_dot` bit for bit; exit 77 where the reference is not ggml's AVX2 build. Registered in the top-level build (also as `iq_single_parity_off` with `STRATA_CPU_SINGLE_EXACT=0`) and in `sycl/`. - `docs/DETAILS.md`: a paragraph with the switch and the numbers below. ## How it was checked Ryzen 7 3800X (Zen 2, AVX2, no AVX-512), Ubuntu 26.04.1, g++ 15.2 (top-level CPU build) and icpx 2026.1 (SYCL build). - `iq_single_parity`: 0 bitwise mismatches (default and `STRATA_CPU_SINGLE_EXACT=0`); `iq_avx2_parity`: 0 failures. - `iq_avx2_parity --bench --dispatch`, 1 token, one thread, engine path, ms per expert (gate/up): IQ3_S 0.497 -> 0.365, IQ3_XXS 0.343 -> 0.322; down unchanged (0.095). - In the SYCL engine on an Arc A770 (Coder IQ1_M, 8K prompt, 256 tokens, the engine's `pool multi` stats): gate/up 15.2-15.4 -> 14.0-14.2 ms/round, CPU pool call about -3%, output ids identical in every run. End-to-end decode did not change measurably there (the pool is ~30 ms of a ~110 ms round). **Not built: CUDA and HIP** (no toolchains on this machine). The files are shared CPU code; nothing CUDA/HIP-specific is touched, but please build those before merging. Measured on Zen 2 only. ## Extra Notes An earlier version also had an exact IQ4_NL down kernel; it measured ~2% slower in the engine, so it is not in this PR. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
Sur le site
Liens install, modèles, releases.