#6412 [Engine] Scope opt-in IQ2_S and IQ3_S Metal prompt kernels for M5 Ultra
open · @ain-soph · 0 コメント · GitHub で見る
本文
### Area [Engine] ### What do you want to do? Would a narrow opt-in change to Metal's native IQ2_S and IQ3_S expert prefill kernels be useful for the Apple M5 Ultra? In the fork, the gate/up kernel uses the existing shared quantization tables for these formats, reuses decoded block headers across the eight sub-blocks, and adjusts tile sizes. I would start with the expert path and validate it separately from dense GEMM and any change that reorders floating-point accumulation. The default should remain unchanged. On one M5 Ultra (96 GB), for the 1,024-row MoE microbenchmark, the fork measurements were 3,534.5 → 3,036.7 µs for IQ2_S and 4,365.8 → 3,090.6 µs for IQ3_S. The experiment's combined prefill kernel arm (including dense changes) read the 115,856-token AD-4.27 prompt at 2,409.8 vs 2,220.3 tok/s. The end-to-end number belongs to that combined arm; the expert changes need their own end-to-end result on current main before submission. I would check the IQ formats against the reference, run a paired prefill A/B, and verify greedy answers and checkpoints with any precision-changing experiment disabled. Does the scope fit the Metal engine, and is there a preferred device or quantization gate for a first PR? ### What do you do today instead? Run the existing Metal expert kernels with the pack resident.
関連リンク
インストール・モデル・リリースへのサイト内リンク。