Issues / #3006
#3006 [Bug]: The Mac (Metal) engine ignores a request's sampling: temperature, top_p, top_k, min_p, seed and the penalties do nothing, logit_bias is dropped, and every answer is the greedy one
open · @oceanio · 0 comentarios · En GitHub
Setup & installServer & APIAMD / HIPNVIDIA / CUDAModels & quants
Descripción
### What happened
On the Mac engine every request decodes greedily, whatever it asks for. `temperature`, `top_p`, `top_k`, `min_p`,
`seed`, `repetition_penalty`, `frequency_penalty` and `presence_penalty` reach the engine (they are on its `GEN` line)
but change nothing. Different seeds at temperature 1.0 give the same text, and it is the temperature-0 text.
`logit_bias` is dropped too: a token banned with -100 is still picked.
Three consequences:
1. **Sampling settings do nothing on a Mac.**
2. **The model card's defaults from 0.1.45 (#2685) do not take effect on a Mac.** The server now sends
temperature 0.7, top_p 0.8, top_k 20 and presence_penalty 1.5 (instruct) when the request and the config give
none. The Mac engine answers those requests greedily, so Mac users still get the greedy decoding that #2685 was
about.
3. **`logit_bias` is dropped on Metal.** `Verifier::set_logit_bias` (`src/core/verify.cpp`) returns before it stores
the bias when `le_ < g_->n_layers`. The Metal init returns before `le_` is set to the layer count, so `le_` stays
-1 and every bias is treated as belonging to a later stage. The request gets no error.
Expected: the same behaviour as the PC engine. The settings act in the CUDA sampler's order, a fixed seed gives the
same text again, another seed gives another text, temperature 0 (or none, without the card's defaults) is greedy, and
a banned token is never picked.
The cause is that the Mac decode window always takes its own argmax (`argmax_part` / `argmax_final` in the window's
head), and `Verifier::run`'s Metal branch returns that token. The host-side sampling that the CUDA path runs after
its window is not ported. Nothing in the log or the response says that the settings were ignored.
#### Reproduction
Built from upstream `main` 477ac13c plus #2895 (the 0.1.45 Mac build fix; without it the Mac engine does not link).
I used the official two-file Qwen3.8-Flash-Next IQ3_S (SC117's abliterated build), resident, as written by
`./setup.sh`. Start the server on a free port with the model's config, then send the same chat request with
different sampling:
```
python3 serve/server.py --engine strata --config strata-<model>.json --port 8197 &
ask() { # the same prompt, thinking off, 40 tokens; $1 = extra JSON fields
curl -s http://127.0.0.1:8197/v1/chat/completions -H 'Content-Type: application/json' -d '{
"model": "strata", "max_tokens": 40, "chat_template_kwargs": {"enable_thinking": false},
"messages": [{"role": "user", "content": "Write one sentence about the sea."}] '"$1"' }' |
python3 -c 'import json,sys; print(json.load(sys.stdin)["choices"][0]["message"]["content"])'
}
ask ', "temperature": 0, "presence_penalty": 0' # greedy
ask ', "temperature": 0' # the card adds presence_penalty 1.5 (#2940)
ask ', "temperature": 1.0, "seed": 1'
ask ', "temperature": 1.0, "seed": 2'
ask ', "temperature": 1.0, "seed": 3'
ask ', "temperature": 1.5, "top_k": 50, "top_p": 0.95, "min_p": 0.01, "seed": 7, "presence_penalty": 0'
ask '' # no sampling fields: the model card's defaults
ask ', "temperature": 0, "presence_penalty": 0, "repetition_penalty": 1.5, "frequency_penalty": 1.0'
ask ', "temperature": 0, "presence_penalty": 0, "logit_bias": {"760": -100}' # 760 = "The", the greedy first token
```
(760 is the first token of the greedy answer with this model's tokenizer; use the first token of your greedy answer.)
Every line prints the same sentence.
#### Measured
M5 Max (128 GB), macOS 26.7.1, Qwen3.8-Flash-Next GSQ-RCO IQ3_S, resident, `--spec 2` with MTP,
`--max-context 262144`, 2026-10-18. These are the answers to the requests above, all in one server run:
| Request (besides the prompt) | Answer |
|---|---|
| `temperature 0, presence_penalty 0` (greedy) | The vast, endless sea stretches to the horizon, its rhythmic waves whispering ancient secrets to the shore. |
| `temperature 0` (the card adds presence_penalty 1.5, #2940) | the same |
| `temperature 1.0, seed 1` | the same |
| `temperature 1.0, seed 2` | the same |
| `temperature 1.0, seed 3` | the same |
| `temperature 1.5, top_k 50, top_p 0.95, min_p 0.01, seed 7` | the same |
| no sampling fields (the model card's defaults: temperature 0.7, top_p 0.8, top_k 20, presence_penalty 1.5) | the same |
| `temperature 0, repetition_penalty 1.5, frequency_penalty 1.0` | the same |
| `temperature 0, logit_bias {"760": -100}` (760 = "The", the greedy answer's first token) | the same, starting with "The" |
The engine reported `0.1.45-metal` in `/v1/status`. An earlier run that day, with nearly the same requests, gave
the same: every answer equal to the greedy one.
With a port of the CUDA sampler's chain to the Mac engine (on the host, over the window's logits in unified
memory; I will open it as a PR once #2895 is merged), the same requests give a different answer for every row but
the first. Seed 1 gives the same text each
time it is sent, seeds 2 and 3 give other texts, and the banned token is not picked ("Beneath the shimmering
surface, ...").
At the engine protocol, a test I wrote for that port (`tools/metal_sampling_test.py`, run against my Metal branch
without the port) fails 12 of 18 checks. Seeds 1 and 2 give the greedy text. 100 one-token draws at temperature 1.0, top_k 40,
top_p 0.95 all give the greedy token, which has probability 0.81 on those logits; 15 of the 100 picks differ from what
a reference sampler written from the CUDA kernels draws with the same seeds. Greedy with `penalty_repeat 1.5` gives the same tokens as
without it. A token banned by `logit_bias` is still picked first.
#### Not tested
- The PC engines (CUDA, HIP, SYCL): not built here. The reference behaviour is read from their code
(`src/kernels/cuda/sampler.cu`, `Verifier::run`) and from the test's reference sampler written from it, not run.
- Other Macs and chips, models that stream their experts from the SSD, other packs (Q2_0, IQ3_XXS, Unsloth's),
vision requests. The cause is in code that all of them share, so I expect the same result.
- Thinking on. The requests above use thinking off; the card's thinking defaults (temperature 1.0, top_p 0.95,
top_k 20) should be ignored the same way.
### Strata version, GPU, OS
0.1.45 (`main` 477ac13c + #2895, built from source) · Apple M5 Max 128 GB (Metal) · macOS 26.7.1
### Engine log
Not attached. Nothing in it mentions sampling: the engine prints no line about the settings it ignored.
En el sitio
Enlaces a install, modelos, releases.