Pull requests / #2609
#2609 Metal: faster decode GEMVs on M1 / M2 (wider K-quant and IQ4_XS loads, tuned tiles), bitwise the same output
open · @PiMatyja · 0 comments · View on GitHub
BenchmarksNVIDIA / CUDAModels & quantsWindows
Description
## Summary Wider loads for the Q4_K, Q5_K, Q6_K and IQ4_XS decoders and a tuned tile choice for the dense decode GEMVs, on M1 and M2 only (Apple GPU family 7 and 8). On an M2 Max with IQ3_XXS, decode is about 11% faster and the output is bitwise the same. M3 and newer build exactly what they build today. ## What changed - `metal/kernels/gemv_wide.metal` (new): the wide decoders, reusing `gemv.metal`'s templates under new kernel names. - `metal/src/engine/window.cpp`: on M1 / M2, adds that file and picks a tuned tile for 1-3 token windows. The tile never changes the summation order. If a tuned kernel is missing, too big for threadgroup memory, or the tensor isn't 16-byte aligned, it falls back to today's tile. - `metal/src/mtl_rt.h` / `.mm`: `max_threadgroup_memory()` and `has_function()`. - `metal/tests/gemv_parity.cpp`: checks every new kernel bitwise against its stock twin; new `--sweep`. `STRATA_METAL_GEMV=0` turns it off. `units.metal`, `quant.metal` and `gemv.metal` are untouched, so no other kernel changes on any GPU. ## Extra Notes M2 Max (30-core GPU), 64 GB, macOS 26.4.1, IQ3_XXS resident (with #2338), greedy, 3 rounds, decode tok/s: | | GEMV=0 | default | | --- | ---: | ---: | | 400 tokens, code | 35.3 | 39.1 (+10.8%) | | 400 tokens, story | 35.2 | 38.7 (+10.0%) | | 256 tokens after ~1K | 35.0 | 39.0 (+11.5%) | | 256 tokens after ~4K | 34.6 | 38.5 (+11.2%) | Same text in both arms on all four prompts. Prompt speed unchanged. `gemv_parity` and `gemv_parity --sweep` pass bitwise on the M2 and on an M4 Max, and the other parity tests match main. Not tested on an M1.
Related on this site
Quick links to related pages.