#3773 [Engine] q36: MTP drafts for sampled requests behind STRATA_Q36_MTP_SAMPLED (stacked on #3678)
open · draft · @baptisterajaut · 0 Kommentare · Auf GitHub
BenchmarksServer & APIMulti-GPUAMD / HIPNVIDIA / CUDAModels & quants
Beschreibung
## Problem and result On the q36 path a request with a temperature above 0 or a repeat / frequency / presence penalty decodes one token at a time even with the MTP block loaded (`spec = E.mtp_d > 0 && !smp.needs_logits() && !adm`), so the drafts only serve greedy requests. With `STRATA_Q36_MTP_SAMPLED=1` a sampled request alone on the engine gets the speculative steps too, and its text keeps the sampled distribution. Without the variable nothing changes. Stacked on #3678: the first commit is #3678 as it stands (8e7742e8), the second (83084811) is the fp16 staging fix I posted there (https://github.com/Niko1221/Strata/pull/3678#issuecomment-6349879312, without it the default fp16 cache does not start), and only the last commit (ea58bfcf) is this change. The stack is there because the switch pays only with an attention that reads the cache once per KV head, which #3678 brings for `--kv int8`; on v0.2.2's attention it loses from 16K on (numbers below). ## Related discussion #3354: the server fills in qwen35moe's model-card sampling (temperature 1.0, presence_penalty 1.5) for a request that names none, so served requests report `drafts accepted 0 of 0`; with this variable they draft. #3678 (the quantized caches and their GQA decode kernel) and #2464 (q36 decode falling with context). ## Approach The drafts stay the MTP block's greedy proposals. `spec_step` takes an optional `draw(r)` callback: row r's token is the request's own sampler over row r's logits, with drafts 0 .. r - 1 appended to the history so the penalties see them, instead of the row's argmax. A draft is kept while the draw equals it and the first draw that differs is the step's token, so every emitted token is a draw of the target distribution: a greedy draft is a point mass, for which the exact-match rule is rejection sampling, as `generate.cpp:14926` already does for its point-mass drafts. Each emitted token costs one draw of the request's rng, so with the slot tables fixed the same seed gives the same text with and without drafts. The serve loop keeps the drawn token across the step (its row's logits are gone) and passes the callback only for sampled requests; greedy requests and the batch window are untouched. ## Validation Environment: RTX 3090, 2x Xeon Gold 6226, 64 GB, CUDA 13.2, Debian 12; Qwen3.8-35B-A3B-Distill Q8_0 (empero-ai, with its MTP layer) in hybrid mode (87-95 of 256 experts per layer in VRAM), `--max-context 131072 --mtp 2 --prefill auto` (chunk 2048), through `serve.server`. **Passed:** - Default path, on v0.2.2 + this change: with the variable unset, the same texts as v0.2.2 as shipped on 3 prompts x 300 tokens, greedy, top_k 1 and sampled (temperature 0.6, presence 0.5, seed 7), 9 of 9 identical, with `STRATA_Q36_FIXED_SLOTS=1`. - Same seed, same text, which is what covers the accepted drafts: with the variable set, the same 9 texts as v0.2.2 as shipped (fixed slots); on this stack, the same 9 comparisons with the variable unset and set give identical texts under `--kv int8` (1271 of 1680 drafts kept in those answers) and under the fp16 default. Since every emitted token is one draw of the same rng over the same logits, equal texts at equal seeds mean the drafts do not change what is drawn. - Distribution, first verify row only: 400 answers of 2 tokens to one open prompt (temperature 1.0, top_k 50, a fresh seed each), v0.2.2 as shipped against the variable set, chi-square 13.1 on 21 degrees of freedom, p ~ 0.9. The second token is drawn on the first verify row, before any draft is kept, so this checks the verify rows' logits, not the acceptance. - Decode speed on this stack, variable unset then set, default (dynamic) slots, thinking off, temperature 0.6 / top_p 0.95 / top_k 20, 1600 tokens at most, two runs per arm: - `--kv int8`, 60K prompt: 81.1 / 84.9 -> 86.7 / 88.4 tok/s; 2K prompt: 99.3 / 99.9 -> 118.9 / 121.5 tok/s. - fp16 default, 60K prompt: 33.6 / 34.0 -> 26.9 / 26.7 tok/s; 2K prompt: 87.7 / 91.4 -> 100.0 / 101.3 tok/s. - 57-59% of the drafts are kept. - These speed runs drop the page cache before each start and run the server under `numactl --interleave=all`. In hybrid mode the pinned expert copy lands on whichever node has free pages at load; on this box, whose UPI link runs at half width, the same binary decoded 2-3x slower from one start to the next depending on that placement (`numa_maps`: 82-86% of the pinned pages on node 1 against 73-81% on node 0). Interleaving puts it at 50/50 in every arm. - On v0.2.2's attention (v0.2.2 + this change, placement not controlled, so read these as indicative): 2K +10 to +18%, 16K -19% (45.1 / 45.0 -> 37.2 / 35.7), 60K -26% (15.1 / 15.1 -> 11.1 / 11.1). **Failed:** none observed. **Not run:** - HIP and SYCL builds: `q36_engine.cu` is part of the HIP build and I have no AMD toolchain. The change is host C++ in that file, no kernel code. - `--kv fp8` and `--kv q4`, `--batch` > 1 (the batch window does not take this path), and thinking-on answers past 1600 tokens. ## Limits The variable is process-wide, not per request. It loses at long context wherever decode attention reads the cache once per query head (fp16 cache here, v0.2.2's fp32), so it should go with `--kv int8` until that attention shares the KV reads. With the default dynamic slots the verify rows move the experts, so the same seed's text drifts from the one without drafts as rounding does; the chi-square check covers the first verify row only, the accepted drafts rest on the same-seed identity with fixed slots.
Mehr auf der Site
Links zu Install, Modellen, Releases.