Issues / #2537
#2537 [Bug]: Metal: IQ2_XS (GSQ-RCO) fails to load — 6 IQ1_M expert tensors have no Metal path; oversized wrap exits instead of falling back to streaming
open · @cuijs2026 · 0 commentaires · Sur GitHub
BenchmarksSetup & installServer & APIAMD / HIPNVIDIA / CUDAModels & quants
Description
### What happened
The GSQ-RCO **IQ2_XS** file cannot run on a Mac: the Metal engine dies during load, and after working around that it dies again on a Metal buffer limit. IQ2_XS is not offered by `metal/setup_mac.py` (`MAC_MODELS`), so this is about manual installs of a size the PC path supports (CUDA/HIP have IQ1_M kernels) but the Metal port does not.
**Step 1 — IQ1_M has no Metal path.** With the stock files from ISTA-DASLab's repo (pinned revision `ed59f92`), the engine exits while loading shard 1:
```
strata serve: the prompt path allocates its own buffers (too few cache slots to borrow)
ref: unknown type 29
```
Type 29 = IQ1_M. The file is mixed-precision; 6 expert tensors use it:
```
blk.8.ffn_gate_exps.weight IQ1_M [2560, 640, 512]
blk.8.ffn_up_exps.weight IQ1_M [2560, 640, 512]
blk.13.ffn_gate_exps.weight IQ1_M [2560, 640, 512]
blk.13.ffn_up_exps.weight IQ1_M [2560, 640, 512]
blk.37.ffn_gate_exps.weight IQ1_M [2560, 640, 512]
blk.37.ffn_up_exps.weight IQ1_M [2560, 640, 512]
```
`metal/src/ref_quant.cpp` (`block_bytes`) prints `ref: unknown type 29` and `std::exit(2)`; `metal/kernels/moe.metal` has no `T_IQ1_M` instantiation either (`GU_SET` covers Q2_0/IQ3_XXS/IQ2_XXS/IQ2_XS/IQ2_S/IQ3_S/IQ4_XS/K-quants/Q8_0). The CPU kernels note the same gap: `src/kernels/cpu/iq_avx512.cpp` says *"IQ1_M stays on ggml-cpu (no shipped model has IQ1_M expert rows)"* — this IQ2_XS file does ship them, so the assumption no longer holds for the GSQ-RCO IQ2_XS release. CUDA has it (`src/kernels/cuda/iq_kernels.cu` `dq_iq1_m`), so the same file runs on PC.
**Step 2 — single-buffer wrap limit.** As a workaround I converted those 6 tensors offline to Q8_0 (dequantized with ggml's `dequantize_row_iq1_m`, re-quantized with the q8_0 reference rule; all other tensors copied byte-identical; pack rebuilt with `tools/iq_pack.py`). With `STRATA_METAL_STREAM=0` (as on a resident Q2_0) the larger shard 1 (43,472,687,392 bytes) then fails to map as one MTLBuffer:
```
strata serve: the prompt path allocates its own buffers (too few cache slots to borrow)
mtl: wrap 43472699392 failed
```
`Context::wrap` (`metal/src/mtl_rt.mm`) exits(4) instead of falling back. `STRATA_METAL_STREAM=1` avoids it (experts stream from the SSD), which is presumably why `setup_mac.py` chooses stream by size — but a wrapped start on an oversized file should either fall back to streaming or fail with a message that says so.
### What I expected
- Either the Metal port runs IQ2_XS as-is (add IQ1_M to `ref_quant.cpp` + the moe/prompt expert kernels, like CUDA), or
- `setup_mac.py` offers IQ2_XS with the conversion done at setup time, or at least fails with "this size needs IQ1_M, which Metal does not have" instead of a bare `ref: unknown type 29`.
- A wrap that exceeds Metal's buffer limit should fall back to streaming (or say to set `STRATA_METAL_STREAM=1`) rather than `exit(4)`.
### Workaround (works, measured below)
1. Rewrite shard 1 with the 6 IQ1_M tensors as Q8_0 (`ref`-exact IQ1_M dequant → q8_0 reference quantize; ~+4.2 GB file size; verified: 1223 tensors, names/order unchanged, every other tensor byte-identical, `iq_pack.py` rebuilds cleanly).
2. `"env": {"STRATA_METAL_STREAM": "1"}` in the config.
3. IQ2_XS does not need its own MTP layer: `mtp_pack.py --experts q2_0` output is shared (the Metal loader reads `<mtp dir>/../mtp-q2_0.gguf`).
### Measurements (M4 Pro 64 GB, macOS, engine 0.1.44.1-metal)
Both sizes, same machine, `tools/strata_bench.py` (thinking off, medians):
| | IQ2_XS (streamed) | Q2_0 (resident) | change |
|---|---:|---:|---:|
| chat decode tok/s | 26.6 | 33.1 | Q2_0 +24% |
| 4K prompt tok/s | 211.8 | 264.7 | Q2_0 +25% |
| 16K prompt tok/s | 223.4 | 273.5 | Q2_0 +22% |
| agent decode tok/s | 25.7 | 32.1 | Q2_0 +25% |
Note the comparison also mixes two differences: IQ2_XS streams (its pack does not wrap) while Q2_0 stays resident, and Q2_0's `STRATA_METAL_STREAM=0` is the faster path on this Mac. Long-context recall (`tools/needle_bench.py`, 32K, depths 10/50/90): **3/3 found for both sizes**. Memory with IQ2_XS running: engine RSS 35.4 GB, no swap growth, 128K context (`--max-context 131072`) held a 124K-token prompt at 246.8 tok/s prefill.
Also worth noting for the "6 IQ1_M tensors" case: they are only in layers 8/13/37, gate+up; downs are Q2_0 like every other layer — a per-tensor fallback (dequant those rows once at load) would also cover it.
### Engine log
```
[mapping/expert-cache lines from startup]
ref: unknown type 29
```
and, after the workaround:
```
[strata metal: decode window ...]
mtl: wrap 43472699392 failed
```
Sur le site
Liens install, modèles, releases.