#5927 [Engine] a pack without tokenizer.json is silently encoded with the qwen35 pre-tokenizer (DeepSeek digits split into characters)
closed · @TurtleFlyRU · 1 コメント · GitHub で見る
BenchmarksSetup & installServer & APINVIDIA / CUDAModels & quantsDocumentationLinux
本文
Problem: a DeepSeek-V4.1-Flash pack whose tokenizer directory has no `tokenizer.json` is served with the **qwen35** pre-tokenizer without any warning, so every prompt is encoded with another model's text-splitting rule. For V4.1 that changes digit grouping (DeepSeek splits digits into 1–3-digit tokens, qwen35 into single characters). Observed on our host: bare numbers come back truncated or spaced (`504` → `5`, `940` → `94`, `1234567` → `1 2 3 4 5 6 7`), numeric tails of copied strings are mangled, and needle-in-a-haystack found **0 of 6** (3 depths × 8K and 32K). Same pack with a `pre` label: all six found.
This is the "wrong rule without an error" case that 37e91bf wrote into `strata_tokenizer.py`: the guard fires for an **unknown** label, but `"qwen35"` is a known label, so the missing-label path at `serve/server.py:10511` passes it through silently:
```python
tok = ST.Tokenizer(tokens, merges, types, tcfg.get("pre") or "qwen35", tcfg.get("special_ids") or None)
```
Second symptom, same root: `arch_from_tokenizer()` (#4137, b18292d) reads the same `tcfg["pre"]`, so with no `tokenizer.json` the family cannot be inferred either. We got the Qwen defaults and `HTTP 400 {"error":{"type":"invalid_request_error","message":"empty prompt"}}` from a V4.1 config until we added `"arch": "deepseek41"` by hand.
Environment:
- `Niko1221/Strata` at **d799195** (2026-11-03), Xolotl tree, `serve/server.py` run on its own.
- Linux 7.0.0-42-generic, RTX 3090 24 GB + RTX 3060 12 GB, driver 580.178.04, Python 3.14.4 (project venv).
- Model: DeepSeek-V4.1-Flash-1017 XQ v4 GGUF (163 GB) + Engram tables on NVMe, context 262144, `--kv-fp4`, DSpark sidecar.
- Tokenizer dir: `vocab.json`, `merges.txt`, `token_type.json`, and the 91-byte marker `chat_template.jinja` — exactly what `tools/v41/serve/export_tokenizer.py` writes. No `tokenizer.json`.
- Config: hand-written, `"arch"` absent (the shape #4137 describes).
Reproduce:
1. The tokenizer half needs no GPU and no server:
```python
import json, sys
sys.path.insert(0, "tools"); sys.path.insert(0, ".")
import strata_tokenizer as ST
from pathlib import Path
tp = Path("/path/to/pack/tokenizer")
vocab = json.loads((tp / "vocab.json").read_text())
tokens = [None] * len(vocab)
for t, i in vocab.items():
tokens[i] = t
merges = (tp / "merges.txt").read_text().split("\n")
types = json.loads((tp / "token_type.json").read_text())
for pre in ("qwen35", "joyai-llm"):
tk = ST.Tokenizer(tokens, merges, types, pre)
print(pre, [tokens[i] for i in tk.encode("504")], [tokens[i] for i in tk.encode("1234567")])
```
Expected: the same ids under both labels (they agree on our corpus sample). Actual:
```text
qwen35 ['5', '0', '4'] ['1', '2', '3', '4', '5', '6', '7']
joyai-llm ['504'] (id 20996) ['123', '456', '7'] (ids 6895, 18009, 25)
```
`"deepseek"` (the V3 label, matched by `pre.startswith("deepseek")`) gives the same ids as `joyai-llm` on digits, punctuation, code and dates — `decode(encode(x))` round-trips in every case, so this is the encoder, not the decoder.
2. The serving half: start `serve/server.py` with that config and ask the model to echo a number.
```bash
curl -s localhost:8150/v1/chat/completions -H 'content-type: application/json' -d '{
"model":"x","temperature":0,"max_tokens":24,
"messages":[{"role":"user","content":"Repeat exactly, no explanation: 504"}]}'
```
Expected: `504`. Actual with no `tokenizer.json`: `5` (`finish_reason: stop`). With a `tokenizer.json` carrying `"pre": "deepseek"`: `504`.
Evidence (all measured here, same pack, same engine binary, only the pre-tokenizer label differs):
| Probe | no `tokenizer.json` (qwen35) | with `pre` label |
|---|---|---|
| echo `504` / `940` / `696` / `1234567` | `5` / `94` / `6` / `1 2 3 4 5 6 7` | `504` / `940` / `696` / `1234567` |
| `Товар стоит 1299 рублей, скидка 15%` | — | exact |
| needle 8K, depths 10/50/90 | **0 of 3** | **3 of 3** |
| needle 32K, depths 10/50/90 | **0 of 3** | **3 of 3** |
| prompt tokens, 32K needle probe | 32 248 | 30 088 |
| decode / prefill (`tools/throughput.py`, 8K prompt) | 30.1 / 379 tok/s | 31.9 / 380 tok/s |
The last two rows are why this is easy to miss: the wrong rule makes prompts *longer* and slightly slower, not broken, and every answer still reads like text.
Expected behavior: a pack that carries no `pre` label should not be encoded with another model's rule. The reasoning already in `strata_tokenizer.py` applies unchanged — "encoding with the qwen35 rule would change every prompt without any error."
Suggested fix (smallest first):
1. `tools/v41/serve/export_tokenizer.py` writes `tokenizer.json` next to the other four files with the label it already tests against (`"joyai-llm"`, `pre_pattern_source` pointing at `DEEPSEEK3_PATTERNS`). Then the generic server both encodes correctly and can infer the family through #4137, and the pack is self-describing for gateways and hand-written configs.
2. If a pack still arrives without it, `serve/server.py` should not fall back to `qwen35` silently: one line naming the missing file (`<pack>/tokenizer/tokenizer.json`), or refusal when the vocabulary is clearly DeepSeek's (`<|begin▁of▁sentence|>`, `<|DSML|>`, `<|deepseek_image|>` are all single ids).
Happy to send a PR for (1) if that is the shape you want — it is a few lines in `export_tokenizer.py` plus a test that the written file round-trips through `arch_from_tokenizer`.
Reproduction status: reproduced locally (all numbers above are from this host).
What I did not test: the repo test suite (`pytest` is not installed in our environment, so no automated run); other packs that go through `export_tokenizer.py` (`tools/glm`, `tools/m31` write the same four files, but I did not run them); and any engine other than `build-xolotl/ds41_generate`. Reading `export_tokenizer.py` shows it writes only `vocab.json`, `merges.txt`, `token_type.json` and the marker `chat_template.jinja` — for the GLM/M31 exporters that statement is **unconfirmed static analysis**.
Note on the documented path: `docs/ds4/V41_VISION.md` serves V4.1 through `tools/v41/serve/serve_ds41.py`, whose `Ds41Tokenizer` forces `pre="joyai-llm"` and is therefore unaffected. The failure above is specific to `serve/server.py` run on its own, i.e. the same situation as #4137.
関連リンク
インストール・モデル・リリースへのサイト内リンク。