Pull requests / #2931

#2931 Metal: hc matrices of any format quantized to Q8_0 at load; gemv for IQ3_XXS and IQ2_S

open · @tolstenko · 0 comentários · No GitHub

BenchmarksAMD / HIPNVIDIA / CUDAModels & quantsDocumentation

Descrição

## What it changes

Two related fixes in `metal/`, both hit while running **uncensored fine-tune GGUFs of Qwen3.8-Flash-Next on Apple Silicon** (files whose hc/attention tensors are quantized differently from the original and the Unsloth releases). `metal/` only; the CUDA/HIP builds are untouched.

### 1. The hyper-connection read assumed BF16/Q8_0/F32 hc tensors

`Window::init`'s HC-Q8 step (metal/src/engine/window.cpp) only quantized `hc_` matrices that were already **BF16**; Q8_0 and F32 were read natively by `gr_down_8_8`/`gr_down_q8_8_8`/`gr_up*`. A fine-tune that re-quantized those tensors to **IQ3_S (down/inject) and IQ4_NL (up)** made the host pick the plain kernels, which read the bytes of a different format. The logits came out NaN on the first decode; the sampler's all-NaN path returned token **2147483647** (the `bi = 0x7fffffff` sentinel in sample.metal's `argmax_part`), the server surfaced it as `IndexError: token id 2147483647 is outside the vocabulary`, and the engine then segfaulted in `iq3s_grid` inside `mtp_encode` when the drafter gathered the embedding of that bogus token.

The fix is in the loader, not new kernels: the HC-Q8 step now takes `hc_` matrices of **every** quantized format (skips only Q8_0/F32, which the file's own kernels already read, and `*_norm`), dequantizes from the tensor's own type, and quantizes to Q8_0 once at load — the same treatment BF16 tensors already got. `gr_down_q8`/`gr_up_q8` then handle them unchanged.

### 2. `gemv.metal` was missing IQ3_XXS and IQ2_S

`moe.metal` instantiates both (expert gate/up), but plain `gemv` did not — so a model with an IQ3_XXS shared-expert gate (`ffn_gate_shexp`) or an IQ2_S attention projection failed with `no function gemv_18_16_1_4_1` / `gemv_22_16_1_4_1` (exit 3) before its first token. Added `GV_SET(T_IQ3_XXS)` and `GV_SET(T_IQ2_S)` to the default `GEMV_TYPES`.

## Measured (M2 Max, 96 GB unified memory, macOS 26.6.2)

With **RVN Uncensored IQ3_S** (88.9 GB merged GGUF, experts streamed from the SSD, `--spec 2` + MTP):

- Before: first request → `token id 2147483647 is outside the vocabulary (248320)`; direct CLI run → SIGSEGV in `iq3s_grid` (stack: `mtp_encode` ← `mtp_draft`).
- After: `/health` → `"loaded": true`; completions return real text; **decode 9.0-10.4 tok/s** (51-token story in 4.9 s), MTP drafting 19/32 accepted. The HC-Q8 step logs `290 matrices, 680 MB, made in 222 ms`.
- Same run, warm prompt (25 tokens): 107 ms/token prompt, 95.7 ms/token decode.
- The OrcaRouter Uncensored IQ3_XXS pack (2 shards, 85.2 GB) hits the same two failures on stock main; with this patch its remaining pipeline (metal_repack + iq_pack --compat-bf16) reaches a loaded server as well.

The HC-Q8 conversion is a one-time per-load cost (222 ms); resident and streaming modes both quantize only the hc matrices, everything else keeps the file's own kernels, so the validated packs' paths are unchanged. `STRATA_HC_Q8=0` still selects the exact-BF16 read for files whose hc tensors are BF16.

## What I did not test

- CUDA/HIP: the change is entirely under `metal/` (and the two files it touches are Mac-only paths).
- Intel Macs; other chips than the M2 Max.
- A full quality benchmark of the fine-tunes (the replies are coherent; the needles and KL numbers in docs/MAC.md were not re-run).

Happy to add anything else the review wants measured.

Fixes the failure mode behind: running a non-validated quantization of the model on the Mac engine — the engine refused with a missing-function error (clear) or silently produced NaN tokens (not clear) depending on which tensor hit first.

No site

Links install, modelos, releases.