#5818 [Engine] Mac (M2 Ultra): Unsloth UD-IQ1_M (v3.1) generates garbage text; llama.cpp on the same files is coherent
closed · @peculiar-ragdoll · 3 评论 · 去 GitHub 看
BenchmarksSetup & installServer & APINVIDIA / CUDAModels & quantsDocumentationWindows
说明
### Area
[Engine] (speed, kernels, models, the server and API)
### Desktop app, Terminal or source?
Source (START-HERE.bat, setup.sh or a build)
### What happened
Every reply from the Metal engine with Unsloth's `UD-IQ1_M` set (Qwen3.8-Flash-Next, Unsloth Dynamic v3.1) is gibberish. Greedy (`temperature 0`, `max_tokens 200`), setup's config with pictures off, the start of the three replies (the text is in `reasoning_content`; `content` is null):
| Prompt | Start of the reply |
| --- | --- |
| Write a Python function that merges overlapping intervals in a list of [start, end] pairs. | `moymmymmymmmmmmymmymm mmmmmmmmo *mmmmm m *mmimmmmyym *mmimmmm****m*mmmmm*m**m*mmxiidmmm*m` |
| Explain in three short paragraphs why the sky is blue. | `y*ym:*mmohim mmmymmmom**mmmmmmmya *mmm mm m*:mmmm·:**m*mm*mm*m*:m:ya*:m**yamyaamaya*myym` |
| List the first ten prime numbers and say what makes 2 special among them. | `mmymmmmmmymmmmmm :mmmmm m*mmmmmmm ***mimmmmmmmmm****mmmmmm*m**mmyayaimmyaayamyaymamommmm` |
Expected: a normal answer.
- With and without the MTP draft layer (`--spec 2 --mtp ...`), the three replies are character-for-character identical.
- With `STRATA_METAL_F16=1` the text changes but is still gibberish. On top of F16, these all gave the same text as F16 alone: MTP, `STRATA_METAL_PROMPT_CHUNK=4096`, `STRATA_PREFILL_CPU_SHARE=0` and `--prefill auto:16384`.
- With pictures on (`--vision`) I did not save the reply text, but the draft acceptance was just as low (below).
The draft acceptance is very low: MTP `--spec 2` accepted 134 of 1926 drafts over 8 requests, and DFlash2 (`--draft dflash2`, the stock pack and the tuned pack) about 8%. Decode with MTP is slower than without it (about 36 vs 53 tok/s).
**The same files work in llama.cpp.** `llama-server` (Homebrew, version 0.7.0, build 11669, commit 620d2225c) on the same shard 1, `-c 4096 -ngl 99 --jinja`, greedy, answers "why is the sky blue" correctly: *"Sunlight looks white, but it is made of many colors. When it enters Earth's atmosphere, tiny gas molecules scatter the light. Shorter wavelengths, like blue and violet, scatter much more strongly..."*
**Checks that passed:**
- Setup checked all three shards against its pinned SHA-256s.
- `metal_repack.py --verify`: 1223 tensors, SHA-256 identical to the source shards.
- `expert_parity --gu {IQ2_XXS,IQ2_XS,IQ2_S} --dn {Q2_0,IQ4_NL} --ntok {1,3}`: all 12 combinations PASS. These are the expert types this file uses: gate/up IQ2_XXS/IQ2_XS/IQ2_S, down Q2_0 in 46 layers and IQ4_NL in 2.
- `gemv_parity` (all formats): ALL PASS, including BF16, Q8_0, IQ4_XS, Q5_K and Q6_K.
**What differs from UD-Q4_K_XL (the Unsloth set measured on Mac in docs/MAC.md).** I read the dense tensor types from the GGUF headers; I did not test UD-Q4_K_XL, UD-Q2_0 or UD-IQ2_XS myself.
| Tensor | UD-Q4_K_XL | UD-IQ1_M (also UD-Q2_0 and UD-IQ2_XS, except attn_qkv in IQ2_XS) |
| --- | --- | --- |
| attn_q / attn_k / attn_v / attn_output / attn_gate, output | Q8_0 | Q6_K |
| attn_qkv | Q8_0 | Q5_K |
| token_embd | Q8_0 | IQ4_XS |
| hc_attn_inject / hc_ffn_inject, ssm_alpha / ssm_beta, ple_conv1d | F32 | BF16 |
| ple_key / ple_value | Q8_0 | BF16 |
### Smallest steps or command that reproduce it
```sh
./setup.sh --setup --model UD-IQ1_M --family unsloth --gguf-dir <dir with the 3 UD-IQ1_M shards> --vision yes --no-start --yes
./run-unsloth-ud-iq1_m.sh
curl -s localhost:8080/v1/chat/completions -H 'Content-Type: application/json' \
-d '{"messages":[{"role":"user","content":"Explain in three short paragraphs why the sky is blue."}],"temperature":0,"max_tokens":200}'
```
Engine command line written by setup (paths shortened):
```
strata --serve --pack Strata-data/packs/unsloth-ud-iq1_m --native Strata-data/metal/unsloth-ud-iq1_m-metal.gguf
--ple-gguf <models>/UD-IQ1_M/Qwen3.8-Flash-Next-UD-IQ1_M-00003-of-00003.gguf
--native-dense-gguf Strata-data/metal/unsloth-ud-iq1_m-metal.gguf --expert-profile Strata/data/expert-profile.bin
--expert-cache 64 --prefill auto --spec 2 --mtp Strata-data/mtp/rt --max-context 32768 --mmap-experts --vision
```
Config `env`: `STRATA_METAL_STREAM=0`. The model is resident (no expert streaming).
### Reproduction status
Reproduced locally
### Strata version
Strata 0.2.13.1 + 152 commits (main c1ff07f1, 2026-11-11) · engine 0.1.59 (`strata 0.1.59-metal`). On v0.2.11 (9f7d977d) the MTP acceptance was just as low (4-11% per request) and decode with MTP was slower than without it. I did not save reply text on that version.
### Graphics card, memory and system
Apple M2 Ultra · 64 GB unified memory · macOS 27.2 (26B123) · `iogpu.wired_limit_mb=57344`
### Model, size and settings
Qwen3.8-Flash-Next Unsloth UD-IQ1_M (unsloth/Qwen3.8-Flash-Next-GGUF, `UD-IQ1_M/`) · 32K context · reply text captured with pictures off; with pictures on, the same low draft acceptance
### Logs
From the engine log on c1ff07f1 (MTP `--spec 2`, pictures on), first two requests:
```
strata generate: experts via mmap (--mmap-experts; the GGUF shards in place, no experts.bin)
strata metal: decode window on Apple M2 Ultra: .../metal/unsloth-ud-iq1_m-metal.gguf wrapped (39.2 GB), 32896 cells, PLE on, ready in 1583 ms
strata metal: MTP draft layer from .../mtp/mtp-q2_0.gguf (0.89 GB, 9 norms folded), draft head over 106299 tokens
strata serve: prompt 116 tokens = 0 reused + 116 read in 842 ms (137.7 tok/s), 256 generated in 7294 ms (35.1 tok/s), drafts accepted 13 of 245, 1 checkpoints
strata serve: prompt 107 tokens = 0 reused + 107 read in 584 ms (183.3 tok/s), 256 generated in 7147 ms (35.8 tok/s), drafts accepted 17 of 244, 1 checkpoints
```
I can attach the full `strata-unsloth-ud-iq1_m.log` and the pack's `conversions.json`, or run a per-layer dump if you name the tool.
本站相关内容
相关页面的快捷入口。