#6536 [Engine] fix #3354: keep default API requests greedy under MTP, gate sampled drafting behind STRATA_Q36_MTP_SAMPLED
open · @6700xtdude-gif · 0 comentarios · En GitHub
Server & APIAMD / HIPNVIDIA / CUDAWindows
Descripción
## Problem Closes #3354. When a `qwen35moe` instance runs with MTP enabled (`--mtp`), an OpenAI/Anthropic API request that carries no sampling parameters still gets the model card's defaults injected (`temperature=1.0`, `presence_penalty=1.5`). The engine only starts MTP drafting when the request does not need logits (`!smp.needs_logits()`), so the silent defaults turned MTP off and the log showed `drafts accepted 0 of 0`, roughly halving decode speed. ## Result - A default API request on an MTP model now stays greedy, so MTP drafting keeps working and the caller gets the same text as before. - A caller who sends explicit sampling parameters still gets exactly those parameters. - The opt-in `STRATA_Q36_MTP_SAMPLED=1` enables draft verification for sampled requests, with the acceptance loop sampling correctly instead of re-sampling verified tokens. ## Approach ### Server (`terminal/serve/server.py`, `terminal/serve/server_parts/http_api.py`) - Added `request_asks_sampling(*layers)` to detect explicit sampling parameters in a request. - Added a `has_mtp` property to `Service`, true when the engine was started with `--mtp` or reports MTP in its info. - In `Service._run()` and `_props()`, model-card defaults are not injected when `has_mtp` is set and the request asks for no sampling, so the request stays greedy. Explicit sampling, or `temperature=0`, takes precedence as before. ### Engine (`engine/src/qwen35/q36_engine.cu`) - Added the `STRATA_Q36_MTP_SAMPLED=1` environment toggle. - Draft verification is enabled in both the single-stream speculative path and the batch MTP path when the toggle is set. - Added `need_sample` state tracking across the speculative acceptance loop so verified tokens are not sampled twice. ### Tests (`terminal/serve/test_model_card_sampling.py`) - Default request under `has_mtp` emits zero sampling keys. - Explicit sampling under `has_mtp` is preserved. - Explicit `temperature=0` stays greedy with no penalties. - `request_asks_sampling` helper behaviour. ## Validation Passed on this machine (Windows 11 Pro, Python 3.13.17): ```text uv run --with jinja2 --with regex python -m unittest serve.test_model_card_sampling -v Ran 12 tests in 0.613s OK ``` Not run: the CUDA change in `q36_engine.cu` was not compiled here — this host has an AMD RX 6700 XT and no CUDA toolchain. That file needs a build plus one MTP decode comparison on maintainer hardware before merge. Not measured: no throughput numbers are claimed in this PR. ## Notes - Based on `3142b66`. As of 2026-11-22, `git merge-tree --write-tree origin/main HEAD` merges cleanly into `6e38f58` (v0.2.18). - Two older PRs cover nearby ground: #3908 (different flags) and #3773 (draft, stacked on #3678, uses the same toggle name). This one is a self-contained single commit against current `main`; close it if you would rather land either of those. - Prepared with AI assistance; the test run above was executed on this machine.
En el sitio
Enlaces a install, modelos, releases.