Pull requests / #2609
#2609 Metal: faster decode GEMVs on M1 / M2 (wider K-quant and IQ4_XS loads, tuned tiles), bitwise the same output
open · @PiMatyja · 0 comentários · No GitHub
BenchmarksNVIDIA / CUDAModels & quantsWindows
Descrição
## Summary Wider loads for the Q4_K, Q5_K, Q6_K and IQ4_XS decoders and a tuned tile choice for the dense decode GEMVs, on M1 and M2 only (Apple GPU family 7 and 8). On an M2 Max with IQ3_XXS, decode is about 11% faster and the output is bitwise the same. M3 and newer build exactly what they build today. ## What changed - `metal/kernels/gemv_wide.metal` (new): the wide decoders, reusing `gemv.metal`'s templates under new kernel names. - `metal/src/engine/window.cpp`: on M1 / M2, adds that file and picks a tuned tile for 1-3 token windows. The tile never changes the summation order. If a tuned kernel is missing, too big for threadgroup memory, or the tensor isn't 16-byte aligned, it falls back to today's tile. - `metal/src/mtl_rt.h` / `.mm`: `max_threadgroup_memory()` and `has_function()`. - `metal/tests/gemv_parity.cpp`: checks every new kernel bitwise against its stock twin; new `--sweep`. `STRATA_METAL_GEMV=0` turns it off. `units.metal`, `quant.metal` and `gemv.metal` are untouched, so no other kernel changes on any GPU. ## Extra Notes M2 Max (30-core GPU), 64 GB, macOS 26.4.1, IQ3_XXS resident (with #2338), greedy, 3 rounds, decode tok/s: | | GEMV=0 | default | | --- | ---: | ---: | | 400 tokens, code | 35.3 | 39.1 (+10.8%) | | 400 tokens, story | 35.2 | 38.7 (+10.0%) | | 256 tokens after ~1K | 35.0 | 39.0 (+11.5%) | | 256 tokens after ~4K | 34.6 | 38.5 (+11.2%) | Same text in both arms on all four prompts. Prompt speed unchanged. `gemv_parity` and `gemv_parity --sweep` pass bitwise on the M2 and on an M4 Max, and the other parity tests match main. Not tested on an M1.
No site
Links install, modelos, releases.