#6659 [Engine] Speculative decoding drafts argmax for sampled requests, so almost every drafted token is rejected
open · @xvcezrqpbxtpzblrt · 0 comments · View on GitHub
BenchmarksSetup & installServer & APIMulti-GPUNVIDIA / CUDAModels & quantsWindows
Description
Thanks for the report. Please read the contributing guide first and search open and closed issues before filing.
**Write only what you saw.** Where a claim below is from reading code rather than a run, it is marked as static analysis in Reproduction status.
> This report is a static source review of the official tree. It was not reproduced on a running build; Reproduction status below says so explicitly.
### Area
[Engine] (speed, kernels, models, the server and API)
### Desktop app, Terminal or source?
Source (the engine's MTP / speculative-decoding path)
### What happened
For any **sampled** request (temperature > 0, i.e. the default mode of almost every chat UI and API client), speculative decoding in the current default configuration spends draft compute but commits almost nothing. The reason is a mismatch between how the draft is produced and how it is verified:
* The MTP drafter, in its default path, always proposes the **argmax** token of the draft head, independently of the request's sampling parameters (`engine/src/core/mtp.cpp:1409-1415`).
* The verify window for a sampled request keeps a drafted token **only when it exactly equals the token the target independently sampled for that row** (`engine/src/include/strata/core/coupled_draft.hpp:3-6`, and the commit loop at `engine/src/program/generate.cpp:22174-22175`). The project's own comment states that the output is then "a function of the seed alone, whatever the drafts are: drafts only decide how many tokens a window yields."
So the acceptance probability of a draft is roughly p(argmax) under the request's temperature/top-p chain — low at temperature ~1 over a large vocabulary. Most drafted rows are rejected, the verify window degenerates to a plain one-token step, and the per-window draft pass (VRAM for the drafter, draft head + chain work) is pure overhead for sampled users. Greedy requests are the only ones that get a real speedup from `--spec` by default.
The engine already contains a complete fix for this: **coupled draft sampling** (`STRATA_SPEC_COUPLED` / `--coupled-draft`), which draws the draft with the target's own chain and the *same* Philox uniform that will verify it, so the two picks agree far more often while the output text stays exactly the same; and **probabilistic draft acceptance** (`STRATA_SPEC_PROB`), which uses the draft's full distribution q with rejection sampling and is exact with respect to the target distribution p. **Both are off by default.**
Source evidence (verbatim):
`engine/src/core/mtp.cpp:1397-1415` — default draft is argmax; the distribution-aware paths are guarded by `coupled_rec_`:
```cpp
if (coupled_rec_) {
// coupled draft sampling: the target's chain and Philox draw for the row that will verify this draft
// (counter = this cell + 1, from its step record), penalties over the ring; T == 1 (full layer)
if (m_q_ != nullptr) // STRATA_SPEC_PROB: a draw from q with its own stream, and q's list for the verifier
spec_draft_sample(head_logits_, (int) nv, sub ? dvocab_ : nullptr, sub ? dinv_ : nullptr, (int) n_vocab_,
cparams_, cring_, kCoupledHistCap, coupled_j_, step, cscratch_, out_ids_, probs_, m_q_,
spec_gate_pick(), spec_draft_temp_scale(), cs);
else
coupled_draft_sample(head_logits_, (int) nv, sub ? dvocab_ : nullptr, sub ? dinv_ : nullptr, (int) n_vocab_,
cparams_, cring_, kCoupledHistCap, coupled_j_, step, cscratch_, out_ids_, probs_, cs);
return true;
}
if (argmax_rows_wanted()) {
argmax_rows(head_logits_, T, (int) nv, arg_scratch_, out_ids_, cs);
} else {
SamplerParams sp;
sp.greedy = true;
sp.temperature = 0.0f;
sample_tokens(head_logits_, T, (int) nv, nullptr, 0, sp, out_ids_, cs);
}
```
`engine/src/include/strata/core/coupled_draft.hpp:1,3-6` — exact-match acceptance and the "drafts only decide how many tokens" consequence:
```cpp
// include/strata/core/coupled_draft.hpp - COUPLED DRAFT SAMPLING (STRATA_SPEC_COUPLED=1, off by default).
...
// Under sampling the verify window samples row t of a window at position pos0 as Philox(seed, pos0 + t) over the
// sampler chain (penalties -> top_k -> top_p -> min_p -> temperature -> inverse-CDF pick; verify.cpp run(),
// sampler.cu), and a draft is kept only when it EQUALS that row's pick. The output is therefore a function of the
// seed alone, whatever the drafts are: drafts only decide how many tokens a window yields.
```
`engine/src/include/strata/core/coupled_draft.hpp:8-12` — the in-tree fix and its property (same output, better agreement):
```cpp
// The MTP draft layer used to propose its ARGMAX. In coupled mode it SAMPLES instead, with the target's own chain
// (the request's top_k / top_p / min_p / temperature and penalties over the same history) and the SAME uniform the
// target will draw for the row that verifies the draft. When the draft's distribution is close to the target's,
// the two inverse-CDF picks from one uniform agree far more often than an argmax agrees with a sample; the target's
// pick - and so the text - does not change, because verification is still exact-match against the target's sample.
```
`engine/src/include/strata/core/coupled_draft.hpp:57-79` — both switches default to off; `coupled_rec_` itself defaults to false at `engine/src/include/strata/core/mtp.hpp:265`:
```cpp
inline bool coupled_draft_state() {
static bool on = [] {
const char* e = std::getenv("STRATA_SPEC_COUPLED");
return e != nullptr && *e != '\0' && std::strcmp(e, "0") != 0;
}();
return on;
}
...
inline bool spec_prob_env() {
static const bool on = [] {
const char* e = std::getenv("STRATA_SPEC_PROB");
return e != nullptr && *e != '\0' && std::strcmp(e, "0") != 0;
}();
return on;
}
```
`engine/src/include/strata/core/spec_prob.hpp:1,3-9` — the probabilistic path is also off by default and is exact:
```cpp
// include/strata/core/spec_prob.hpp - PROBABILISTIC DRAFT ACCEPTANCE (STRATA_SPEC_PROB=1, off by default).
...
// Speculative rejection sampling (Leviathan et al. / Chen et al.) for the MTP drafts of a SAMPLED request. Under
// sampling the verify window used to accept a draft only when it EQUALS the token the target sampled for that row
// (exact match). For a draft that is the head's argmax - a point mass - that is already the standard rule, but it
// throws away what the draft head knows: its whole distribution q. Here the draft layer SAMPLES its draft d from q
// ...
// p, the target's chain (penalties -> top_k -> top_p -> min_p -> temperature), whatever q is.
```
`engine/src/program/generate.cpp:22174-22175` — the exact-match commit rule in the verify path:
```cpp
int a = merged ? T - 1 : 0; // a merged window commits every row, as the prompt windows do
while (a < T - 1 && window[(size_t) a + 1] == outv[(size_t) a]) ++a;
```
`engine/src/include/strata/core/spec_prob.hpp:64-66` — the project's own measurement note, and the reason a temperature gate belongs in the fix:
```cpp
/// STRATA_SPEC_MIN_TEMP=x: coupled / probabilistic drafting only for requests with temperature >= x (default 0: every
/// sampled request). Measured on the RTX 3060 and 5070: sampled drafting gains at temperature 1.0 and loses below
/// about 0.7, where the draft head's argmax is the better guess; below x the drafter keeps its argmax drafts.
```
### Smallest steps or command that reproduce it
1. Start the engine with a drafter and speculative decoding enabled, e.g. `strata serve --spec 4 --mtp <dir>` (or the equivalent CLI `strata generate --spec 4 --mtp <dir>`), with **no** `STRATA_SPEC_COUPLED` and no `STRATA_SPEC_PROB` in the environment (the defaults).
2. Send a sampled request, e.g. over the serve protocol `GEN 64 temperature=1.0 top_p=0.9 <prompt ids>` (or an API/chat request that uses the default sampling temperature), and watch the speculative counters / tokens-per-second.
3. Repeat the identical request with `STRATA_SPEC_COUPLED=1` (or `--coupled-draft`) and then with `STRATA_SPEC_PROB=1`, same seed and same sampling parameters.
4. Compare accepted draft tokens per verify window and output tokens per second. With the defaults, the accepted-prefix length stays at ~1 token per window for sampled requests; with either switch on it grows, while the emitted text for a fixed seed is unchanged (coupled mode) or distributed exactly as p (probabilistic mode).
An equivalent static check: with no switches set, `coupled_rec_` stays false and the code path at `engine/src/core/mtp.cpp:1409-1415` runs; the acceptance rule at `engine/src/program/generate.cpp:22175` is a plain `==` comparison.
### Reproduction status
Unconfirmed static analysis. The mechanism above is read directly from the source tree; I did not run the engine or benchmark the acceptance rate. All line references are from the source tree at the time of writing and should be re-checked against the release tag before merging.
### Strata version
0.2.19 · engine 0.1.65 (source tree under review, static analysis; not a running-build query)
### Graphics card, memory and system
Not applicable — this is a scheduler logic issue, not tied to a GPU, driver or OS. The source notes that the gain was measured on an RTX 3060 and a 5070 (`engine/src/include/strata/core/spec_prob.hpp:65`), but the argmax-vs-sample mismatch follows from the code path on every device and model that has an MTP drafter.
### Model, size and settings
Any model with an MTP drafter / `--mtp` directory and `--spec >= 2`, any quant, any context size, any temperature > 0. Greedy requests (temperature 0) are the one case where the default rule already accepts argmax drafts, so this report does not apply to them.
### Logs
No runtime logs were collected (static analysis only). The only "evidence text" is the in-source comments and code quoted in "What happened" above, plus the startup warning that appears when probabilistic acceptance is requested but falls back: `engine/src/program/generate.cpp:3498-3499`:
```cpp
std::fprintf(stderr, "strata: STRATA_SPEC_PROB=1 keeps the older acceptance rule in this run (%s). That rule's output "
"distribution is off when the confidence gate fails; STRATA_SPEC_PROB_GATE=top is exact here.\n", why);
```
### Why this is worth fixing for everyone
Sampling is the normal generation mode, not an edge case: every chat UI, API client and creative-writing user sends temperature > 0. For all of them the default speculative path is close to a no-op, while its costs (drafter VRAM, the draft head and chain pass on every window, scheduler complexity) are still paid. This is not a property of a particular GPU, quant or context length: it follows only from "draft = argmax" meeting "verify = exact match against an independent sample".
The fix is model- and hardware-independent because it changes only how the candidate is proposed and accepted, not the target distribution: coupled sampling keeps the output byte-for-byte identical for a fixed seed (the target still verifies by exact match), and probabilistic acceptance is mathematically exact with respect to p. Turning one of them on automatically for sampled requests — using the existing `STRATA_SPEC_MIN_TEMP` gate so the documented low-temperature (~<0.7) regression is avoided — makes `--spec` actually speed up the default user's generation on every machine that can run a drafter. It also removes a confusing situation where the same model is fast with temperature 0 and slow with temperature 1 for no user-visible reason.
### Root cause
The draft and the verifier disagree by construction in the default configuration:
1. `engine/src/core/mtp.cpp:1409-1415`: with `coupled_rec_ == false` (the default, `engiRelated on this site
Quick links to related pages.