Issues / #2713
#2713 let Cascade Lake CPUs (AVX512F/BW/VL/DQ + VNNI, no VBMI) use the AVX-512 i-quant kernels
open · @zhai-hello · 0 commentaires · Sur GitHub
NVIDIA / CUDAModels & quantsDocumentationWindows
Description
My CPU is an Intel i9-10940X (Cascade Lake-X, 14 cores). It has AVX512F, AVX512BW, AVX512VL, AVX512DQ, AVX512CD and AVX512-VNNI. It does **not** have AVX512-VBMI — VBMI came in on Cannon Lake, and Cascade Lake is Skylake-X plus VNNI. `cpu_avx512_ok()` asks for all five bits, so the engine prints > this CPU has no AVX-512: the expert kernels run on AVX-2 (multi-token for the i-quant gate/up rows) and the i-quant gate/up rows run on the AVX-2 kernel. What I want is for the gate to accept CPUs that have F/BW/VL/DQ + VNNI but not VBMI, at least on builds where nothing in those translation units can emit a VBMI instruction. **What it changes:** on the Windows/MSVC build, CPUs of this class (Skylake-X/Brookdale, Cascade Lake-X, Cascade Lake-SP with VNNI) would run the AVX-512 i-quant kernels instead of the AVX-2 ones. **What it leaves alone:** the default path on Ice Lake / Zen 4 and newer stays byte-identical, and the GCC/Clang build keeps asking for VBMI (reason below). ## Why I think it is safe on the MSVC build - `/arch:AVX512` is set on the sources (CMakeLists.txt:1326 for `iq_avx512.cpp` and `nvfp4_avx512.cpp`, :1219 for `expert.cpp`), and MSVC's `/arch:AVX512` covers F/CD/BW/DQ/VL — the compiler will not emit a VBMI instruction from it. - The AVX-512 sources use no VBMI intrinsic. I searched `src/` for the VBMI intrinsics (vpermb / vpermi2b / vpermt2b / vpmovb2m …): the only byte-or-word permute in them is `_mm512_permutexvar_epi16` at iq_avx512.cpp:47, which is AVX512BW. The VNNI intrinsic `_mm512_dpbusd_epi32` is used (expert.cpp:183, nvfp4_avx512.cpp:120) — that is why the gate needs VNNI, and this CPU has it. - nvfp4_avx512.cpp:9 already states the kernels' own requirement: "Needs AVX512F/BW/VL and VNNI - what cpu_avx512_ok() checks before anything here is called." VBMI is extra on top of what the kernels use. - There is already a narrower gate for the same kind of gap: `cpu_avx512bw_ok()`, documented at include/strata/kernels/cpu/expert_layout.hpp:47 as "AVX-512F + AVX-512BW + AVX-512VL + AVX-512DQ (e.g. Skylake-X i9-9920X)". ## Why the gate asks for VBMI anyway On GCC/Clang those same sources are compiled with `-mavx512vbmi` explicitly (CMakeLists.txt:1224, :1339), so on that build a CPU without VBMI can reach a VBMI instruction and take an illegal instruction — which is what the comment at expert_layout.cpp:300 is guarding against. So the fix probably has to be build-aware (relax the bit only where the build cannot emit VBMI), or `tools/check_isa_guard.py` should show the AVX-512 objects are VBMI-free. ## How I checked, and what I have not measured - numpy's runtime CPUID detection on this machine: AVX512F, BW, VL, CD, DQ, VNNI = true; AVX512VBMI = false. - The engine's own check passes OSXSAVE and the XCR0 test and fails only on the VBMI bit. - **I have not measured any speed difference** — I have no build with the relaxed gate. The direction is that the CPU pool does all the missed experts here (`--pcie-frac 0.00`), so a faster CPU kernel should matter more on this box than on a box that streams from PCIe, but I am not claiming a number. ## Machine, and I can test it i9-10940X (Cascade Lake-X, 14C/28T, an ES part, stepping B1), RTX 3090 24 GB, 64 GB DDR4, Windows 10 21H2, engine 0.1.44.1, model Swift 1.5 IQ3_XXS (native pack, `--pcie-frac 0.00`, `--spec-min-p 0.70`). If you put out a test binary with the relaxed gate, I will run the same-day comparison the way docs/COMMUNITY_BENCHMARKS.md asks and post the numbers. Similar reports in other projects, for what they are worth: [ollama #14435](https://github.com/ollama/ollama/issues/14435) (Enable AVX512_VNNI for Cascade Lake CPUs), [ik_llama.cpp #2409](https://github.com/ikawrakow/ik_llama.cpp/issues/2409) (Optimizing the AVX-512 codepath on Intel Cascade Lake).
Sur le site
Liens install, modèles, releases.