#6578 [Engine] The default MTP `draft_vocab.bin` contains only 0.4% of the Arabic tokens: Arabic decode accepts 42% of drafts; adding the missing tokens gives 74% (+38% tok/s)
open · @harment · 0 コメント · GitHub で見る
BenchmarksMulti-GPUAMD / HIPNVIDIA / CUDAModels & quantsLinux
本文
## Summary
The draft head's token subset shipped as `draft_vocab.bin` (106,299 ids, also what `--draft-vocab cjk` / the default picks) contains **33 of the 8,826 Arabic-script tokens** of the Qwen3.8-Flash-Next vocabulary, but **100% of the 65,681 CJK tokens** (62% of the subset). The MTP drafter therefore cannot propose most Arabic tokens, including very common words (` في`, ` من`, ` على`, ` أن`, ` التي`, ` هذا`, ` إلى`). `--draft-vocab` has no Arabic option (`cjk|en|cyrillic|fr|latin`).
Appending the 8,793 missing Arabic-script ids to the existing file (original order kept, nothing removed) and passing it with `--mtp-draft-vocab` raised draft acceptance on Arabic output from **41.7% to 74.1%** and median decode from **88.65 to 122.4 tok/s (+38%)** on the same engine, model and prompts.
## Environment
- Strata 0.2.19, engine 0.1.65 (the baseline was also measured on 0.1.64, see noise floor), Linux x86_64, CUDA 13
- 2x RTX 3090 24 GB, automatic layer split (K=25), 59 GiB RAM
- Model: Qwen3.8-Flash-Next GSQ-RCO IQ3_S (native pack + PLE), MTP drafting (`--mtp <dir>/rt --spec 4 --spec-min-p 0.65`), `--kv k8v4`, `--max-context 262144`
## What I measured
**1. Coverage of the draft subset.** I decoded the byte-level BPE tokens of `vocab.json` (248,320 ids) and classified them by Unicode block (U+0600–06FF, 0750–077F, FB50–FDFF, FE70–FEFF for Arabic script; this also includes Persian/Urdu letters):
| class | tokens in vocab | in `draft_vocab.bin` |
|---|---|---|
| Arabic script | 8,826 | **33 (0.4%)** |
| CJK | 65,681 | 65,681 (100%) |
| other | 173,813 | 40,585 (23.3%) |
On 80 Arabic Wikipedia intro paragraphs (9,868 tokens, approximate BPE implementation) only **19.9%** of tokens are inside the subset; with the Arabic ids added, 99.8%. English prose / Python / C++ / JS are already covered at 96–99%, so the gap is specific to Arabic.
**2. Benchmark.** 60 requests built from those Wikipedia paragraphs (half "continue the paragraph", half "explain the paragraph", Arabic prompts), `temperature 0.2`, `max_tokens 256`, thinking disabled with `chat_template_kwargs: {"enable_thinking": false}`, same running engine settings, only `--mtp-draft-vocab` differs:
| | draft vocab | median decode | mean decode | draft acceptance |
|---|---|---|---|---|
| baseline | 106,299 ids (stock) | 88.65 tok/s | 89.4 | 41.7% |
| with Arabic ids | 115,092 ids | **122.4 tok/s** | 122.7 | **74.1%** |
Noise floor: two identical stock runs on engine 0.1.64 gave 88.95 / 91.1 tok/s and 41.2% / 41.5%. Output stays Arabic (Arabic-letter ratio ≈ 1.0, no empty replies). The log confirms the file was used: `strata mtp: draft head over 115092 tokens (230.5 MiB)` (was 106299 / 212.9 MiB).
## How to reproduce
The script below (appends the Arabic-script ids, keeps the original order) is attached as `make_draft_vocab_ar.py`:
```
python3 make_draft_vocab_ar.py <pack>/tokenizer <mtp>/rt/draft_vocab.bin draft_vocab_ar.bin
strata launch --preset <name> --set mtp-draft-vocab=/path/to/draft_vocab_ar.bin
```
## Suggested fix
- Add Arabic-script tokens to the default subset, or add an `ar` option to `--draft-vocab` (and make Auto pick it from the system language, as it does for the others).
- Possibly check other scripts too (Hebrew, Devanagari, Thai, Greek, ...). I only measured Arabic, so I do not know whether they have the same gap.
- The 65,681 CJK ids take 62% of the draft head (~131 MiB at ~2.1 KB per id). A non-CJK default for users who do not need CJK output would shrink it. I have not tested this yet.
## Caveats
One user, one model, one machine; the BPE tokenization used for the coverage numbers is my own approximation, not the official tokenizer. Output text was not compared byte by byte (drafts are verified by the target model). DFlash2 was not tested; its pack ships the same `draft_vocab.bin` size, but I do not know how its drafter uses the order of the ids.
## Side note (separate from this issue)
`"thinking": false` in the request body is ignored by the server on this model (the reply came back with empty `content` and English `reasoning_content` until `max_tokens`); `chat_template_kwargs.enable_thinking=false` (or `reasoning_effort: "none"`) works. I measured Arabic output only after finding this, so benchmarks that rely on `thinking: false` may be measuring English reasoning rather than the answer.
関連リンク
インストール・モデル・リリースへのサイト内リンク。