Pull requests / #2609

#2609 Metal: faster decode GEMVs on M1 / M2 (wider K-quant and IQ4_XS loads, tuned tiles), bitwise the same output

open · @PiMatyja · 0 comentários · No GitHub

BenchmarksNVIDIA / CUDAModels & quantsWindows

Descrição

## Summary
Wider loads for the Q4_K, Q5_K, Q6_K and IQ4_XS decoders and a tuned tile choice for the dense decode GEMVs, on M1 and M2 only (Apple GPU family 7 and 8). On an M2 Max with IQ3_XXS, decode is about 11% faster and the output is bitwise the same. M3 and newer build exactly what they build today.

## What changed
- `metal/kernels/gemv_wide.metal` (new): the wide decoders, reusing `gemv.metal`'s templates under new kernel names.
- `metal/src/engine/window.cpp`: on M1 / M2, adds that file and picks a tuned tile for 1-3 token windows. The tile never changes the summation order. If a tuned kernel is missing, too big for threadgroup memory, or the tensor isn't 16-byte aligned, it falls back to today's tile.
- `metal/src/mtl_rt.h` / `.mm`: `max_threadgroup_memory()` and `has_function()`.
- `metal/tests/gemv_parity.cpp`: checks every new kernel bitwise against its stock twin; new `--sweep`.

`STRATA_METAL_GEMV=0` turns it off. `units.metal`, `quant.metal` and `gemv.metal` are untouched, so no other kernel changes on any GPU.

## Extra Notes
M2 Max (30-core GPU), 64 GB, macOS 26.4.1, IQ3_XXS resident (with #2338), greedy, 3 rounds, decode tok/s:

| | GEMV=0 | default |
| --- | ---: | ---: |
| 400 tokens, code | 35.3 | 39.1 (+10.8%) |
| 400 tokens, story | 35.2 | 38.7 (+10.0%) |
| 256 tokens after ~1K | 35.0 | 39.0 (+11.5%) |
| 256 tokens after ~4K | 34.6 | 38.5 (+11.2%) |

Same text in both arms on all four prompts. Prompt speed unchanged. `gemv_parity` and `gemv_parity --sweep` pass bitwise on the M2 and on an M4 Max, and the other parity tests match main. Not tested on an M1.

No site

Links install, modelos, releases.