Pull requests / #3119
#3119 serve: temperature 0 stays greedy - no model-card penalties on a greedy request (#2940)
open · @christopherrobertbrooks-tech · 0 comentários · No GitHub
BenchmarksServer & APIAMD / HIPNVIDIA / CUDAModels & quantsDocumentation
Descrição
## Title serve: temperature 0 stays greedy - no model-card penalties on a greedy request Issue: Resolves #2940 ## Summary Since a8091f4d (#2685) a request without its own sampling gets the model card's set merged under it. A request that asks for `temperature: 0` keeps its temperature but still picks up the card's `presence_penalty` 1.5 (Flash-Next when not thinking; Qwen3.6 in both thinking states), so it is not greedy: the engine samples from its logits with the penalty applied. On the Qwen3.6 engine that also means **no MTP drafts on any temperature-0 request**: `spec` needs `!smp.needs_logits()`, and the penalty makes it true. With this change, when the request, the shared Chat settings or the config's `sampling` block (the first one that sets a temperature) asks for temperature 0, the card's penalties are left out. The rest of the card (top_p / top_k, which greedy ignores) stays as before, and a penalty the request or the config sets itself still applies. `/props` reports the same. Requests that do not ask for temperature 0 are unchanged. This is the first of the two suggestions in #2940 (temperature 0 skips the card's set); the second (a config `sampling` block replacing the card instead of merging over it) is left alone. ## What changed - `serve/server.py`: `model_card_sampling(arch, thinking, greedy=False)` drops `presence_penalty`, `frequency_penalty` and `repetition_penalty` when `greedy`; `asks_greedy(*layers)` says whether the first layer that sets a temperature sets 0. `_run` and `/props` pass it. - `serve/test_model_card_sampling.py`: `test_greedy_gets_no_card_penalty` (Flash-Next not thinking, Qwen3.6 both states, the request's own penalty kept, a non-greedy request still gets the card, a config or shared temperature 0) and `test_asks_greedy`. Both fail on 477ac13 and pass with the change; the existing tests in the file are unchanged and pass. - `docs/DETAILS.md`: one sentence after "Greedy is only for who asks for it". ## Extra Notes **Measured** on a Tesla V100-PCIe 32 GB (sm_70), CUDA 12.9, the 0.1.45 engine binary for both arms (only `serve/server.py` differs). Qwen3.6-35B-A3B UD-Q4_K_XL, `--native --max-context 131072 --prefill auto --mtp 2`, every routed expert in VRAM, `reasoning_effort: none`. Same prompt, greedy, 0.1.45 server vs this branch: | request | server | text = true greedy (`presence_penalty: 0` sent) | drafts accepted | decode | | --- | --- | --- | ---: | ---: | | prose, `temperature: 0` | 0.1.45 | no | 0 of 0 | 114.7 tok/s | | prose, `temperature: 0` | this branch | **yes** | 115 of 166 | **166.4 tok/s** | | code, `temperature: 0` | 0.1.45 | no | 0 of 0 | 112.0 tok/s | | code, `temperature: 0` | this branch | **yes** | 260 of 280 | **193.8 tok/s** | `strata-bench-v100/benchmark.py` (temperature 0, fresh prompts filled to each size, 256-token answers, engine `/metrics`, 3 runs, medians), decode tok/s: | prompt tokens | 256 | 1,000 | 4,096 | 16,384 | 32,768 | 65,536 | | --- | ---: | ---: | ---: | ---: | ---: | ---: | | 0.1.45 server (no drafts) | 109.5 | 104.9 | 85.8 | 59.6 | 40.4 | 23.4 | | this branch (drafts, 2,659 of 3,896 accepted) | **156.5** | **156.7** | **116.7** | 57.6 | 34.6 | 19.6 | Prefill is unchanged (1,167 / 1,731 / 1,793 / 1,605 / 1,359 / 984 tok/s). From 16K up the drafts cost on this card, with the default FP32 K/V cache: that is MTP's own trade-off, not this change - 296b59d, from before a8091f4d, measures the same on this rig (154.1 / 154.4 / 114.6 / 56.4 / 33.4 / 17.3). **Tests:** all 50 `serve/test_*.py` files with and without the change: the same results, except `test_responses.NoLeakedSampler.test_server_close_ends_the_sampler`, which fails on and off on 477ac13 as well (a thread count; it failed twice in a row without this change when run alone). **Not tested:** the main (Flash-Next) engine live, AMD, and the batch path (`--batch N`). Only the Python server changes, so there is no backend to build. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
No site
Links install, modelos, releases.