#6525 [Engine] Drop in MTP head replacement trained against quantized model output: +3.7 / +4.4 % decode at --spec 3
open · @paltian · 0 Kommentare · Auf GitHub
BenchmarksSetup & installMulti-GPUAMD / HIPNVIDIA / CUDAModels & quantsDocumentationWindowsLinux
Beschreibung
## Summary [Basalt](https://github.com/jesdga95/basalt), the MIT fork of Strata for Blackwell cards by Jesús Gulfo, ships a self-distilled MTP draft layer for Qwen3.8-Flash-Next. Its `mtp/dense` section uses the same format as Strata's `mtp/rt/dense.bin`, so it can be used in Strata without conversion. Nothing about it is specific to Blackwell, so any Strata user who drafts with the MTP layer should be able to benefit. On my machine, with `--spec 3`, it raises the share of accepted drafts from 72.1 % to 75.1 % and decode speed by 3.7 % at 4k and 4.4 % at 32k. It was faster on all 6 documents. Together with `--spec 3` instead of 2, decode is **+8.5 % at 4k and +7.6 % at 32k** over the default layer with `--spec 2`. ## The file - Basalt's Hugging Face repo serves it on its own as [`convert/mtp/rt/dense.bin`](https://huggingface.co/jesdga/Qwen3.8-Flash-Next-Basalt/blob/main/convert/mtp/rt/dense.bin): 116,099,072 bytes, sha256 `dfe18647e0c95d05302907be88f021590119645a1567fcd770c8d309df852d75`. - `dense-original.bin` next to it (sha256 `c724dc0b…`) is byte-identical to the `mtp/rt/dense.bin` that Strata's setup builds. `experts.bin` and `draft_vocab.bin` in that folder are also byte-identical to Strata's. Only `dense.bin` differs. - Basalt's note (`dense-tuned.json`) says it was "self-distilled on IQ3_S output". I measured it with Unsloth's UD-Q4_K_XL, so it does not seem tied to the quant it was distilled on. - License: the repo distributes it under the Qwen Community License 1.0, the same license as the Qwen weights Strata already ships. How I used it (the default `rt` folder stays untouched): ```sh mkdir -p <data>/mtp/rt-tuned cp <data>/mtp/rt/dense.txt <data>/mtp/rt/draft_vocab.bin <data>/mtp/rt/experts.bin <data>/mtp/rt-tuned/ hf download jesdga/Qwen3.8-Flash-Next-Basalt convert/mtp/rt/dense.bin --local-dir basalt-mtp cp basalt-mtp/convert/mtp/rt/dense.bin <data>/mtp/rt-tuned/ # engine args: --mtp <data>/mtp/rt-tuned --spec 3 ``` ## Measurements - **Hardware:** 2x RTX 5060 Ti 16 GB (PCIe Gen3 x4 and Gen4 x8), Ryzen 7 5700X3D, 96 GB DDR4, Linux, CUDA 13. - **Model and engine:** Unsloth UD-Q4_K_XL. Engine v0.2.17 plus PR #2427, which does not touch the drafter. - **Profile, identical for every arm:** - `--layer-split 11 --trim-stage-weights --pipeline-windows 2 --spec-min-p 0.5 --pcie-frac 0.2` - `--kv int8 --kv-resident 32768 --max-context 262144 --vision` - `STRATA_HC_Q8=1 STRATA_SH_STREAM=1 STRATA_MTP_STAGE0=1` - The only differences are `--mtp` and `--spec`. - **Load:** 3 documents per size from the Strata docs corpus, at about 4k tokens (3.6k–4.9k) and about 32k (31k–37k). Each prompt has a unique tag, so nothing is reused between runs. Thinking off, temperature 0, 512 generated tokens per request. - **Runs:** a fresh server per arm, 2 rounds, so 6 requests per size and arm. All arms ran on the same day. | Draft layer | `--spec` | Decode 4k (tok/s) | Decode 32k (tok/s) | Drafts accepted | Accepted per request | |---|---:|---:|---:|---:|---:| | Strata default | 2 | 51.6 | 48.7 | 79.4 % | 208 | | Basalt's tuned | 2 | 53.0 (+2.7 %) | 49.8 (+2.3 %) | 81.2 % | 210 | | Strata default | 3 | 54.0 | 50.2 | 72.1 % | 272 | | **Basalt's tuned** | **3** | **56.0 (+3.7 %)** | **52.4 (+4.4 %)** | 75.1 % | 278 | | Basalt's tuned | 4 | 54.5 | 51.1 | 69.0 % | 306 | The percentages compare the tuned layer with the default layer at the same `--spec`. - **Per document** (default against tuned at `--spec 3`, mean of 2 rounds): +2.4 / +0.3 / +8.4 % at 4k and +7.8 / +3.6 / +1.8 % at 32k. - **Prompt speed** is the same in every arm: 446–448 tok/s at 4k and 1201–1211 tok/s at 32k. - **Errors:** none in any run. - **Longer windows:** the tuned layer gains more with them (+2.5 % at `--spec 2`, +4 % at `--spec 3`). `--spec 4` is slower again on this machine. ## Suggestion 1. Consider Basalt's `dense.bin` as the `mtp/rt` layer for Qwen3.8-Flash-Next, either as the default or as an option. 2. With it, re-consider the `--spec` default. On this machine 3 beat 2 with both layers. On 4 Nov, with an older version and a different benchmark, `--spec 3` was only ahead on JSON, code and math prompts, not on prose. So the best value may depend on the hardware and the kind of text. Not tested: HIP, SYCL, Metal, other quants, single-card setups.
Mehr auf der Site
Links zu Install, Modellen, Releases.