#6536 [Engine] fix #3354: keep default API requests greedy under MTP, gate sampled drafting behind STRATA_Q36_MTP_SAMPLED

open · @6700xtdude-gif · 0 コメント · GitHub で見る

Server & APIAMD / HIPNVIDIA / CUDAWindows

本文

## Problem

Closes #3354.

When a `qwen35moe` instance runs with MTP enabled (`--mtp`), an OpenAI/Anthropic API request that carries no sampling parameters still gets the model card's defaults injected (`temperature=1.0`, `presence_penalty=1.5`). The engine only starts MTP drafting when the request does not need logits (`!smp.needs_logits()`), so the silent defaults turned MTP off and the log showed `drafts accepted 0 of 0`, roughly halving decode speed.

## Result

- A default API request on an MTP model now stays greedy, so MTP drafting keeps working and the caller gets the same text as before.
- A caller who sends explicit sampling parameters still gets exactly those parameters.
- The opt-in `STRATA_Q36_MTP_SAMPLED=1` enables draft verification for sampled requests, with the acceptance loop sampling correctly instead of re-sampling verified tokens.

## Approach

### Server (`terminal/serve/server.py`, `terminal/serve/server_parts/http_api.py`)

- Added `request_asks_sampling(*layers)` to detect explicit sampling parameters in a request.
- Added a `has_mtp` property to `Service`, true when the engine was started with `--mtp` or reports MTP in its info.
- In `Service._run()` and `_props()`, model-card defaults are not injected when `has_mtp` is set and the request asks for no sampling, so the request stays greedy. Explicit sampling, or `temperature=0`, takes precedence as before.

### Engine (`engine/src/qwen35/q36_engine.cu`)

- Added the `STRATA_Q36_MTP_SAMPLED=1` environment toggle.
- Draft verification is enabled in both the single-stream speculative path and the batch MTP path when the toggle is set.
- Added `need_sample` state tracking across the speculative acceptance loop so verified tokens are not sampled twice.

### Tests (`terminal/serve/test_model_card_sampling.py`)

- Default request under `has_mtp` emits zero sampling keys.
- Explicit sampling under `has_mtp` is preserved.
- Explicit `temperature=0` stays greedy with no penalties.
- `request_asks_sampling` helper behaviour.

## Validation

Passed on this machine (Windows 11 Pro, Python 3.13.17):

```text
uv run --with jinja2 --with regex python -m unittest serve.test_model_card_sampling -v
Ran 12 tests in 0.613s
OK
```

Not run: the CUDA change in `q36_engine.cu` was not compiled here — this host has an AMD RX 6700 XT and no CUDA toolchain. That file needs a build plus one MTP decode comparison on maintainer hardware before merge.

Not measured: no throughput numbers are claimed in this PR.

## Notes

- Based on `3142b66`. As of 2026-11-22, `git merge-tree --write-tree origin/main HEAD` merges cleanly into `6e38f58` (v0.2.18).
- Two older PRs cover nearby ground: #3908 (different flags) and #3773 (draft, stacked on #3678, uses the same toggle name). This one is a self-contained single commit against current `main`; close it if you would rather land either of those.
- Prepared with AI assistance; the test run above was executed on this machine.

関連リンク

インストール・モデル・リリースへのサイト内リンク。