Issues / #3008

#3008 qwen35 (Qwen3.6): --mtp N is silently inert in serve mode unless --batch N --batch-mtp are also passed; setup's own config omits them

open · @FlyoverBlind · 0 commentaires · Sur GitHub

BenchmarksSetup & installServer & APIAMD / HIPNVIDIA / CUDAModels & quantsWindowsLinux

Description

## Summary

On the qwen35 path (Qwen3.6-35B-A3B), `--mtp N` alone does nothing in serve mode. MTP only activates when `--batch N` (≥ 2) **and** `--batch-mtp` are also passed — and even then only for greedy requests. No warning or error is printed when it is silently inactive. The config that setup.py itself generates for qwen36 sets `--mtp` but omits both flags, so the out-of-the-box path runs without MTP and gives no indication.

## What I measured

Same machine, same model (Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf), Strata v0.1.45, single session, decode speed from the `strata serve:` engine log line, steady state:

| serve flags | request | engine log | decode |
|---|---|---|---|
| (no MTP flags) | normal (server default sampling) | — | 34–37 tok/s |
| `--mtp 4` | normal (server default sampling) | `drafts accepted 0 of 0` | 39.0 tok/s |
| `--mtp 4 --batch 2 --batch-mtp` | normal (server default sampling) | `drafts accepted 0 of 0` | 39–42 tok/s |
| `--mtp 4 --batch 2 --batch-mtp` | greedy (temperature 0, penalties zeroed) | `drafts accepted 97 of 106` | 55.9–56.6 tok/s |

The headline is the log line, not the speed: with `--mtp 4` alone the draft model is loaded but `drafts accepted 0 of 0` shows it is never used. With the full flag set and a greedy request it engages (~91% acceptance, ~+45% decode vs. baseline on this machine).

## Where in the code

`src/qwen35/q36_engine.cu`:

- L3883: `E.bmtp = (batch_mtp || (std::getenv("STRATA_BATCH_MTP") ...)) && E.n_sess > 1 && E.mtp_d > 0;`
- L3351–3355: `if (E.bmtp && E.mtp_d > 0) { ... window_mtp(); return; } window_plain();`
- L1192: `int mtp_d = 0; // MTP drafts per speculative step (0: none; greedy requests only)`

So in serve mode `--mtp N` only sets `mtp_d`; without `--batch ≥ 2` → `n_sess == 1` → `bmtp == 0` → every request takes `window_plain()` and `mtp_d` is never consulted. In generate mode `mtp_d` works on its own, which makes the serve-mode asymmetry easy to miss.

There is a second silent gate worth mentioning: even with the flags right, MTP only runs for greedy requests (`needs_logits()` at L3013 — any nonzero temperature or penalty disables it), and server.py's `MODEL_CARD_SAMPLING` defaults for `qwen35moe` (temperature 0.7, presence_penalty 1.5, L6779–6786) make ordinary chat requests non-greedy by default. MTP engaged for me only after zeroing penalties and setting temperature 0.

## Why this looks like a footgun

1. setup.py's own qwen36 family config (`OWN_PATH_FAMILIES["qwen36"]`) sets `mtp: 2` but does not add `--batch` / `--batch-mtp` — the shipped path is MTP-less with no indication.
2. No startup warning: with `--mtp N` in serve mode and `bmtp == 0`, nothing says the draft path is loaded but unused. `drafts accepted 0 of 0` in the log is the only clue.
3. The user-visible flags (`--mtp`) and the activation preconditions (`--batch`, `--batch-mtp`, greedy, server default penalties) live in three different layers; each one alone silently disables MTP.

## Suggested fixes (any one would help)

a) Print a startup warning in serve mode when `mtp_d > 0 && !E.bmtp`: "MTP requested but inactive: needs --batch N --batch-mtp (and greedy requests)".
b) Allow `window_mtp()` with `n_sess == 1` so `--mtp N` works without `--batch`/`--batch-mtp`.
c) Have setup.py add `--batch-mtp` (and a suitable `--batch`) to the generated qwen36 config whenever `mtp` is set.

## Measured on

- Windows 11, RTX 5060 Ti 16 GB, 32 GB RAM
- Strata v0.1.45 (release engine, BUILD.json v0.1.45)
- Model: Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf, sha256 starts with `55983c5a`
- Speeds from the `strata serve:` log line, single request, steady state

## Not tested

- Linux / other GPUs
- The `STRATA_BATCH_MTP` env route (should behave the same by the code path)
- More than 2 sessions
- Whether single-slot (`n_sess == 1`) MTP actually runs correctly if the `n_sess > 1` gate is lifted — I only verified the gate exists, not that the window handles one slot

Sur le site

Liens install, modèles, releases.