#6630 [Engine] Inkling-2-Mini SQ-15 on Metal (M6, 20.5 GiB): every reply is token id 0 ("!"), finish_reason length
open · @vladlensb · 2 Kommentare · Auf GitHub
Beschreibung
### Area
[Engine]
### Desktop app, Terminal or source?
Terminal
### What happened
On Apple M6 (Metal, 20.5 GiB), Inkling-2-Mini SQ-15 answers with only `!` and
`finish_reason: "length"`.
Token id 0 in the o200k vocab is `!`, so the logits are degenerate (zero/NaN) and
`argmax` always picks index 0.
Ruled out: the drafter (same with `--drafting off`), the model file (sha256
`2ea32eed...30737` matches), the tokenizer, expert placement (all 3072 experts in VRAM).
The engine log shows the model is wrapped as **2 overlapping Metal buffers**
(14.35 GB > `maxBufferLength` 14.30 GB, overlap 1.07 GB). That looks like the cause.
### Smallest steps or command that reproduce it
1. `strata launch --family inkling-2-mini --model SQ-15 --context 32k`
2. `POST /v1/chat/completions` with `{"model":"inkling-2-mini-sq-15","messages":[{"role":"user","content":"Hi"}],"max_tokens":64,"temperature":0}`
3. Answer: `!!!!!!!!` with `finish_reason: "length"`.
### Reproduction status
Reproduced locally
### Strata version
Strata 0.2.19 · engine 0.1.65-macos-arm64-metal
### Graphics card, memory and system
Apple M6 · 20.5 GiB unified memory · macOS <version>
### Model, size and settings
Inkling-2-Mini SQ-15 · 32K context · pictures off
### Logs
```
mtl: wrapped 14.35 GB as 2 overlapping buffers (maxBufferLength 14.30 GB, overlap 1.07 GB)
[ink_metal] loaded ... (build=SQ-15, trunk=646, dspark=54, experts=3072, vram_order=3072) in 0.01s (workSet=20.5 GiB, maxBuf=13.3 GiB)
[ink_metal] runner ready: slots=1 ctx=32768 batch=512 dspark=1 vision=0 mpp=1(trunk=0,hot=0) | KV=1184.0 MiB SC=3.7 MiB TotalAlloc=15.64 GiB
```
Mehr auf der Site
Links zu Install, Modellen, Releases.