#6578 [Engine] The default MTP `draft_vocab.bin` contains only 0.4% of the Arabic tokens: Arabic decode accepts 42% of drafts; adding the missing tokens gives 74% (+38% tok/s)

open · @harment · 0 comments · View on GitHub

BenchmarksSetup & installMulti-GPUAMD / HIPNVIDIA / CUDAModels & quantsLinux

Description

## Summary

The draft head's token subset shipped as `draft_vocab.bin` (106,299 ids, also what `--draft-vocab cjk` / the default picks) contains **33 of the 8,826 Arabic-script tokens** of the Qwen3.8-Flash-Next vocabulary, but **100% of the 65,681 CJK tokens** (62% of the subset). The MTP drafter therefore cannot propose most Arabic tokens, including very common words (` في`, ` من`, ` على`, ` أن`, ` التي`, ` هذا`, ` إلى`). `--draft-vocab` has no Arabic option (`cjk|en|cyrillic|fr|latin`).

Appending the 8,793 missing Arabic-script ids to the existing file (original order kept, nothing removed) and passing it with `--mtp-draft-vocab` raised draft acceptance on Arabic output from **41.7% to 74.1%** and median decode from **88.65 to 122.4 tok/s (+38%)** on the same engine, model and prompts.

## Environment

- Strata 0.2.19, engine 0.1.65 (the baseline was also measured on 0.1.64, see noise floor), Linux x86_64, CUDA 13
- 2x RTX 3090 24 GB, automatic layer split (K=25), 59 GiB RAM
- Model: Qwen3.8-Flash-Next GSQ-RCO IQ3_S (native pack + PLE), MTP drafting (`--mtp <dir>/rt --spec 4 --spec-min-p 0.65`), `--kv k8v4`, `--max-context 262144`

## What I measured

**1. Coverage of the draft subset.** I decoded the byte-level BPE tokens of `vocab.json` (248,320 ids) and classified them by Unicode block (U+0600–06FF, 0750–077F, FB50–FDFF, FE70–FEFF for Arabic script; this also includes Persian/Urdu letters):

| class | tokens in vocab | in `draft_vocab.bin` |
|---|---|---|
| Arabic script | 8,826 | **33 (0.4%)** |
| CJK | 65,681 | 65,681 (100%) |
| other | 173,813 | 40,585 (23.3%) |

On 80 Arabic Wikipedia intro paragraphs (9,868 tokens, approximate BPE implementation) only **19.9%** of tokens are inside the subset; with the Arabic ids added, 99.8%. English prose / Python / C++ / JS are already covered at 96–99%, so the gap is specific to Arabic.

**2. Benchmark.** 60 requests built from those Wikipedia paragraphs (half "continue the paragraph", half "explain the paragraph", Arabic prompts), `temperature 0.2`, `max_tokens 256`, thinking disabled with `chat_template_kwargs: {"enable_thinking": false}`, same running engine settings, only `--mtp-draft-vocab` differs:

| | draft vocab | median decode | mean decode | draft acceptance |
|---|---|---|---|---|
| baseline | 106,299 ids (stock) | 88.65 tok/s | 89.4 | 41.7% |
| with Arabic ids | 115,092 ids | **122.4 tok/s** | 122.7 | **74.1%** |

Noise floor: two identical stock runs on engine 0.1.64 gave 88.95 / 91.1 tok/s and 41.2% / 41.5%. Output stays Arabic (Arabic-letter ratio ≈ 1.0, no empty replies). The log confirms the file was used: `strata mtp: draft head over 115092 tokens (230.5 MiB)` (was 106299 / 212.9 MiB).

## Variants tested (same engine 0.1.65, same prompts; Arabic = the 60-request set above, code = 30 programming prompts, thinking off)

The shipped language files follow one pattern: `draft_vocab_en.bin` (40,525 ids, sorted) is the base, and `fr`, `latin`, `cyrillic` add their own tokens on top; the default file is `en` + 65,681 CJK ids. All of them contain the same 33 Arabic ids. I built an Arabic file the same way (`en` + all 8,826 Arabic-script ids = 49,318 ids, sorted) and compared it with the other options:

| draft subset | ids | draft head | Arabic tok/s | Arabic accept | code tok/s | code accept |
|---|---|---|---|---|---|---|
| stock default | 106,299 | 212.9 MiB | 88.65 | 41.7% | not measured | not measured |
| default + Arabic ids | 115,092 | 230.5 MiB | 124.4 | 74.1% | 156.0 | 83.7% |
| default + Arabic + 2.4k frequent code/English ids | 117,503 | 235.3 MiB | 122.1 | 75.0% | 156.35 | 85.1% |
| same, without CJK | 51,588 | 103.3 MiB | 122.7 | 73.5% | 161.35 | 85.5% |
| **`en` + Arabic (the pattern above)** | **49,318** | **98.8 MiB** | 120.2 / 125.0* | 73.5% / 73.7%* | 156.65 / 160.45* | 82.9% / 85.1%* |

\* second measurement of the same file after a clean restart with it installed as the MTP dir's `draft_vocab.bin`.

All improved variants are within the ~2–3% run-to-run noise of each other. So an `en` + Arabic file loses nothing on Arabic or code compared with the larger file, and its draft head is less than half the size of the stock one. Removing the CJK ids did not make decoding measurably faster on this setup (expert cache 8,948 -> 8,963 slots), but it did not hurt either. Order of ids did not matter for the MTP (sorted and unsorted lists both worked).

## How to reproduce

The script below (appends the Arabic-script ids, keeps the original order) is attached as `make_draft_vocab_ar.py`:

```
python3 make_draft_vocab_ar.py <pack>/tokenizer <mtp>/rt/draft_vocab.bin draft_vocab_ar.bin
strata launch --preset <name> --set mtp-draft-vocab=/path/to/draft_vocab_ar.bin
```

## Suggested fix

- Ship `draft_vocab_ar.bin` (= `en` + the Arabic-script ids, 49,318 ids) next to `fr`/`latin`/`cyrillic`, add `ar` to `--draft-vocab`, and make Auto pick it from the system language as it does for the others. I can send the file or a PR if that is welcome.
- Possibly check other scripts too (Hebrew, Devanagari, Thai, Greek, ...). I only measured Arabic, so I do not know whether they have the same gap.
- Not required for this fix: the 65,681 CJK ids are 62% of the default draft head (~131 MiB at ~2.1 KB per id), but removing them gave no measurable speed change here (see the table), only a smaller head.

## Caveats

One user, one model, one machine; the BPE tokenization used for the coverage numbers is my own approximation, not the official tokenizer. Output text was not compared byte by byte (drafts are verified by the target model). DFlash2 was not tested; its pack ships the same `draft_vocab.bin` size, but I do not know how its drafter uses the order of the ids.

## Side note (separate from this issue)

`"thinking": false` in the request body is ignored by the server on this model (the reply came back with empty `content` and English `reasoning_content` until `max_tokens`); `chat_template_kwargs.enable_thinking=false` (or `reasoning_effort: "none"`) works. I measured Arabic output only after finding this, so benchmarks that rely on `thinking: false` may be measuring English reasoning rather than the answer.

[strata_arabic_draft_vocab.zip](https://github.com/user-attachments/files/35238031/strata_arabic_draft_vocab.zip)

Related on this site

Quick links to related pages.