Pull requests / #3301
#3301 [Engine] metal: the request's sampling on the Mac engine (temperature, top_p, top_k, min_p, seed, penalties, logit_bias)
open · draft · @oceanio · 0 comentarios · En GitHub
BenchmarksServer & APIAMD / HIPNVIDIA / CUDAModels & quantsDocumentation
Descripción
## Problem and result On the Mac (Metal) engine every request decodes greedily. `temperature`, `top_p`, `top_k`, `min_p`, `seed` and the penalties reach the engine on its `GEN` line but change nothing, and `logit_bias` is dropped. Since 0.1.45 the server also sends the model card's sampling when a request gives none, and on a Mac that has no effect either. Nothing in the log or the response says so (#3006). With this PR the Mac engine samples as the PC engine does: the same chain and order (logit_bias, penalties, top_k, top_p, min_p, temperature), one Philox draw per token keyed by the seed and the position, so a fixed seed repeats its text and another seed gives another. Temperature 0 (or none) stays greedy and gives the same tokens as before. ## Related discussion #3006 (the bug, reproduced again on 0.2.0-beta; see Validation). A focused fix, so no scope issue first. ## Approach `Verifier::run`'s Metal branch returned the window's argmax. The CUDA path samples on the host after its window; this PR does the same on the Mac, on the CPU, over the window's head logits, which are already in unified memory (no copy). A request that is greedy and has no penalties and no `logit_bias` keeps the window's argmax, so its tokens are exactly the old ones. The MTP drafts stay greedy and a draft is kept when the sampled token equals it, as on the PC. - `metal/src/engine/host_sampler.cpp` / `.h`: the CUDA sampler's chain with the kernels' float/double arithmetic and tie order. 9-46 µs per row on 248,320 logits (M5 Max), so no GPU kernel. - `metal/src/engine/window.cpp`: an accessor for row t of the last window's head logits. - `src/core/verify.cpp`: the sampling after the Metal window (inside `#if defined(STRATA_GPU_METAL)`), and one line in `set_logit_bias` that every backend compiles: `if (le_ >= 0 && le_ < g_->n_layers) return true;`. The Metal init leaves `le_` at -1, which the old test read as "an earlier stage" and so never stored the bias. The CUDA, HIP and SYCL inits set `le_` to `n_layers` when it is -1 (`verify.cpp`, `init`), so after init the condition is the same for them. ## Validation Environment: M5 Max 128 GB, macOS 26.7.1, Command Line Tools 27.0 (Apple clang 21). Qwen3.8-Flash-Next GSQ-RCO IQ3_S (SC117's abliterated build, the official two-file layout), resident, `--spec 2` with MTP, `--mmap-experts`, `--max-context 262144`. Base = `main` 3fa1b524 (0.2.0-beta + 7 commits), built from the same tree. Rebased since onto `main` e88ee39d (2026-10-20; its 5 new commits change no engine, Metal or server file): the 3 commits build again with all Metal targets, `host_sampler_parity` passes again; the runs below are on 3fa1b524. **Passed:** - `ninja -C build-metal` (all Metal targets), at each of the 3 commits: 0 errors, no new warnings. - `build-metal/host_sampler_parity` (CPU): 592 picks, all equal a serial transcription of the CUDA kernels (`sampler_parity.cpp`'s mirror); Philox matches 4,096 draws. - `python3 tools/metal_sampling_test.py --exe build-metal/strata --config <cfg>`: 13 of 13 pass. On the base build the same script fails 11 of 13 (every seed gives the greedy text; penalties and `logit_bias` change nothing). 208 one-token requests drew exactly the token a reference sampler (written in the test from `sampler.cu`) draws from the same dumped logits. - #3006's reproduction through `serve/server.py` (9 requests, the curl script in the issue): base, 9 of 9 print the greedy sentence; this PR, the greedy request prints the same sentence as the base, and the other 8 (seeds 1-3 at temperature 1.0, top_k/top_p/min_p, the card's defaults, penalties, `logit_bias` 760:-100) each print another sentence; the banned first token is not picked. - Greedy unchanged: temperature 0 through the server on a 28-, a 2,257- and a 16,180-token prompt, identical text to the base build. **Failed:** none observed. **Not run:** - CUDA, HIP and SYCL builds (no such toolchains on a Mac). The only line they compile is the `set_logit_bias` test above. Asking for a build on one of them before this leaves draft. - Other Macs and models; Swift / Unsloth packs; batch slots (Metal has none on `main`, #2715). - Not ported: the opt-in draft schemes `STRATA_SPEC_COUPLED`, `STRATA_SPEC_PROB`, `STRATA_SPEC_GUMBEL` (the Mac drafts greedily). ## Limits - Sampled decode is a little slower than greedy: 74.8 / 72.9 / 68.8 against 76.3 / 83.3 / 78.3 tok/s over 3 runs (4 short prompts, 256 tokens, MTP; `--bench`), because fewer drafts are kept (0.70 against 0.72) and the texts differ. The pick itself costs under 0.05 ms a token. - The card's presence penalty now applies on a Mac too: a client that wants the old greedy text sends `temperature: 0` and `presence_penalty: 0` (documented in docs/MAC.md). 🤖 Generated with [Claude Code](https://claude.com/claude-code)
En el sitio
Enlaces a install, modelos, releases.