Pull requests / #1582
#1582 tokenizer: initialize caches once per instance
open · @InB4DevOps · 0 Kommentare · Auf GitHub
BenchmarksSetup & installServer & APIWindowsLinux
Beschreibung
## Summary
Initialize `_piece_ids` and `_bytes_cache` once in `Tokenizer.__init__`, then access those attributes directly.
The previous `self.__dict__.setdefault("_bytes_cache", {})` constructed an empty default dictionary and looked up the cache through `__dict__` for every decoded token, even on hits. `_encode_plain()` used the same pattern per segment.
Cache contents and limits, BPE merge order, special-token handling and byte decoding are unchanged. The two empty dictionaries are allocated at construction rather than first use.
This PR contains only:
- `tools/strata_tokenizer.py`
- `tools/test_strata_tokenizer.py`
No parser optimizations, profiling infrastructure, benchmark files, or native/GPU changes are included.
## Isolated validation
Candidate and untouched baseline both started at upstream `fb58e0dbc8399662c0e47c76578c6e878b14f6cf`. Server, frontend and benchmark-fixture source hashes were identical between checkouts; only the tokenizer and its test file differed.
```text
python -m unittest tools.test_strata_tokenizer serve.test_detok
Ran 15 tests — OK (skipped=5)
```
**10 tests passed; five tests requiring unavailable tokenizer assets were skipped.** Added checks cover:
- Decoding before any encoding, cold and warm, including Unicode/CRLF/NUL and all 256 raw byte tokens.
- Invalid IDs before and after cache use.
- Separate tokenizer instances with different vocabularies, exercising both piece and byte cache independence.
- Concurrent first-use decoding on a shared tokenizer.
Existing tests cover cold/warm/disabled piece caching, cache bounds, synthetic BPE parity and concurrent encoding. `git diff --check` passed.
**Windows correctness and performance validation: pending.**
## Same-day CPU measurements
Intel Core i7-12700KF, Linux 7.0.0-38-generic, glibc 2.39, CPython 3.12.3, regex 2026.9.29, Jinja2 3.1.6. Default scheduling/power policy, no affinity pinning. Seven fresh-process samples per checkout, alternating baseline/candidate order, two warmups per case, profiling disabled.
Deterministic synthetic vocabulary (seed 268, 1,500 merges), 49,664 tokens / 66,048 UTF-8 bytes. Caches are warm; fixture construction, validation and hashing are outside timing. No model or GPU required.
| Workload | Untouched upstream median | PR median | Elapsed reduction |
|---|---:|---:|---:|
| Incremental detokenization | 12.86 ms | 9.85 ms | 23.4% |
| Detokenization + upstream parser + collection | 88.70 ms | 84.93 ms | 4.3% |
| Warm encoding | 3.94 ms | 3.94 ms | Approximately unchanged |
Detokenization ranges: 12.68–13.13 ms upstream, 9.77–10.16 ms candidate. All 42 measured outputs matched fixtures and had matching hashes across checkouts and samples. These are synthetic Python CPU measurements, **not inference-throughput claims**; constructor/startup performance was not benchmarked.
### Reproduction
With `regex` and `jinja2` installed, run this from the root of the baseline and PR checkouts. It uses the repository's existing GPU-free tokenizer fixture and prints one warmed sample of each workload. Repeat in fresh processes and alternate checkouts for comparisons.
```python
import time
from serve.server import Detokenizer
from serve.frontend import OutputParser
from serve.test_detok import synthetic_tokenizer
tok = synthetic_tokenizer(seed=268, n_merges=1500)
code = "def accumulate(values):\n return sum(x * x for x in values)\n# café 日本語 🙂\n"
text = ("The result is correct. café 日本語 🙂\n" + code) * 512
ids = tok.encode(text, parse_special=False)
assert tok.decode(ids, errors="strict") == text
def detokenize():
detok = Detokenizer(tok)
return "".join(detok.push(token) for token in ids)
def pipeline():
detok, parser, chunks = Detokenizer(tok), OutputParser(thinking=False), []
for token in ids:
chunks.extend(e.text for e in parser.feed(detok.push(token)) if e.kind == "content")
chunks.extend(e.text for e in parser.finish() if e.kind == "content")
return "".join(chunks)
for name, fn, expected in (
("encode", lambda: tok.encode(text, parse_special=False), ids),
("detokenize", detokenize, text),
("pipeline", pipeline, text),
):
for _ in range(2):
assert fn() == expected
start = time.perf_counter_ns()
result = fn()
elapsed_ms = (time.perf_counter_ns() - start) / 1e6
assert result == expected
print(name, elapsed_ms)
```
Mehr auf der Site
Links zu Install, Modellen, Releases.