Pull requests / #2348

#2348 metal: the M1 Max prompt kernels - a select with a device load in the…

open · @demetree · 0 comentarios · En GitHub

BenchmarksAMD / HIPNVIDIA / CUDAModels & quantsDocumentation

Descripción

… arm loses the threadgroup

The two prompt kernels (qsa_attn_mm, qsa_flash) fill their single-simdgroup matmul2d right inputs with

    rk[i] = id >= 0 ? half(float(kq[row * QD + k]) * float(ksc[...])) : half(0.0f);

and that shape - a select with the device load in the arm - loses the threadgroup on an M1 Max: every dispatch of the two kernels fails with
kIOGPUCommandBufferCallbackErrorInnocentVictim, deterministically, and they run only under MTL_SHADER_VALIDATION=1. The engine builds this library at start, so nothing is served.

The row is now clamped with max(id, 0), the load is unconditional, and the select is on the value. For an id >= 0 the value is what it was; for an id < 0 it was discarded either way, so the results do not change.

Measured on an M1 Max (macOS 26.6.2, Metal compiler 32023): qsa_prompt_parity ALL PASS - qsa_attn_mm vs qsa_attn rel L2 3.09e-04 (pos0 0) and 3.56e-04 (pos0 30000), qsa_flash 3.72e-04 and 4.22e-04, qsa_score_mm worst rel 2.66e-06 (T 96, within budget and with 2051 selected cells a token); before the change the test faulted on its first dispatch. Through the server, on the reporter's streaming config (Q2_0, --expert-cache 64, prefill auto, spec 2): a 9,940 -token prompt prefilled in 53.4 s (186 tok/s), 24 tokens decoded in 3.0 s (7.9 tok/s), no GPU error.

The compile failure that hid this - qsa_flash's cooperative_tensor copy, "cannot initialize a parameter of type 'void *' with an rvalue of type 'const void *'" - is already fixed in main (#2323); this change is the runtime half of the same two kernels' M1 Max bring-up.

What it leaves alone: the COOP_K4 path (a function constant: compiled out of the M1/M4 pipelines), qsa.metal, and every host file; no CUDA, HIP or SYCL backend builds these kernels. docs/MAC.md records the M1 Max with the measured tok/s.

## Title
<!-- If Applicable, reference the GitHub issue -->
Issue: Resolves # 

## Summary
<!-- Quick Summary of changes -->

## What changed
<!-- Specifics on files changed, and what changes were made there -->

## Extra Notes
<!-- Any extra notes, delete if there are none -->

En el sitio

Enlaces a install, modelos, releases.