Pull requests / #3301

#3301 [Engine] metal: the request's sampling on the Mac engine (temperature, top_p, top_k, min_p, seed, penalties, logit_bias)

open · draft · @oceanio · 0 comentarios · En GitHub

BenchmarksServer & APIAMD / HIPNVIDIA / CUDAModels & quantsDocumentation

Descripción

## Problem and result
On the Mac (Metal) engine every request decodes greedily. `temperature`, `top_p`, `top_k`, `min_p`, `seed` and the
penalties reach the engine on its `GEN` line but change nothing, and `logit_bias` is dropped. Since 0.1.45 the server
also sends the model card's sampling when a request gives none, and on a Mac that has no effect either. Nothing in the
log or the response says so (#3006).

With this PR the Mac engine samples as the PC engine does: the same chain and order (logit_bias, penalties, top_k,
top_p, min_p, temperature), one Philox draw per token keyed by the seed and the position, so a fixed seed repeats its
text and another seed gives another. Temperature 0 (or none) stays greedy and gives the same tokens as before.

## Related discussion
#3006 (the bug, reproduced again on 0.2.0-beta; see Validation). A focused fix, so no scope issue first.

## Approach
`Verifier::run`'s Metal branch returned the window's argmax. The CUDA path samples on the host after its window; this
PR does the same on the Mac, on the CPU, over the window's head logits, which are already in unified memory (no copy).
A request that is greedy and has no penalties and no `logit_bias` keeps the window's argmax, so its tokens are exactly
the old ones. The MTP drafts stay greedy and a draft is kept when the sampled token equals it, as on the PC.

- `metal/src/engine/host_sampler.cpp` / `.h`: the CUDA sampler's chain with the kernels' float/double arithmetic and
  tie order. 9-46 µs per row on 248,320 logits (M5 Max), so no GPU kernel.
- `metal/src/engine/window.cpp`: an accessor for row t of the last window's head logits.
- `src/core/verify.cpp`: the sampling after the Metal window (inside `#if defined(STRATA_GPU_METAL)`), and one line in
  `set_logit_bias` that every backend compiles: `if (le_ >= 0 && le_ < g_->n_layers) return true;`. The Metal init
  leaves `le_` at -1, which the old test read as "an earlier stage" and so never stored the bias. The CUDA, HIP and SYCL
  inits set `le_` to `n_layers` when it is -1 (`verify.cpp`, `init`), so after init the condition is the same for them.

## Validation
Environment: M5 Max 128 GB, macOS 26.7.1, Command Line Tools 27.0 (Apple clang 21). Qwen3.8-Flash-Next GSQ-RCO IQ3_S
(SC117's abliterated build, the official two-file layout), resident, `--spec 2` with MTP, `--mmap-experts`,
`--max-context 262144`. Base = `main` 3fa1b524 (0.2.0-beta + 7 commits), built from the same tree.
Rebased since onto `main` e88ee39d (2026-10-20; its 5 new commits change no engine, Metal or server file): the
3 commits build again with all Metal targets, `host_sampler_parity` passes again; the runs below are on 3fa1b524.

**Passed:**
- `ninja -C build-metal` (all Metal targets), at each of the 3 commits: 0 errors, no new warnings.
- `build-metal/host_sampler_parity` (CPU): 592 picks, all equal a serial transcription of the CUDA kernels
  (`sampler_parity.cpp`'s mirror); Philox matches 4,096 draws.
- `python3 tools/metal_sampling_test.py --exe build-metal/strata --config <cfg>`: 13 of 13 pass. On the base build the
  same script fails 11 of 13 (every seed gives the greedy text; penalties and `logit_bias` change nothing). 208
  one-token requests drew exactly the token a reference sampler (written in the test from `sampler.cu`) draws from the
  same dumped logits.
- #3006's reproduction through `serve/server.py` (9 requests, the curl script in the issue): base, 9 of 9 print the
  greedy sentence; this PR, the greedy request prints the same sentence as the base, and the other 8 (seeds 1-3 at
  temperature 1.0, top_k/top_p/min_p, the card's defaults, penalties, `logit_bias` 760:-100) each print another
  sentence; the banned first token is not picked.
- Greedy unchanged: temperature 0 through the server on a 28-, a 2,257- and a 16,180-token prompt, identical text to
  the base build.

**Failed:** none observed.

**Not run:**
- CUDA, HIP and SYCL builds (no such toolchains on a Mac). The only line they compile is the `set_logit_bias` test
  above. Asking for a build on one of them before this leaves draft.
- Other Macs and models; Swift / Unsloth packs; batch slots (Metal has none on `main`, #2715).
- Not ported: the opt-in draft schemes `STRATA_SPEC_COUPLED`, `STRATA_SPEC_PROB`, `STRATA_SPEC_GUMBEL` (the Mac drafts
  greedily).

## Limits
- Sampled decode is a little slower than greedy: 74.8 / 72.9 / 68.8 against 76.3 / 83.3 / 78.3 tok/s over 3 runs
  (4 short prompts, 256 tokens, MTP; `--bench`), because fewer drafts are kept (0.70 against 0.72) and the texts
  differ. The pick itself costs under 0.05 ms a token.
- The card's presence penalty now applies on a Mac too: a client that wants the old greedy text sends
  `temperature: 0` and `presence_penalty: 0` (documented in docs/MAC.md).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

En el sitio

Enlaces a install, modelos, releases.