Pull requests / #1582

#1582 tokenizer: initialize caches once per instance

open · @InB4DevOps · 0 comments · View on GitHub

BenchmarksSetup & installServer & APIWindowsLinux

Description

## Summary

Initialize `_piece_ids` and `_bytes_cache` once in `Tokenizer.__init__`, then access those attributes directly.

The previous `self.__dict__.setdefault("_bytes_cache", {})` constructed an empty default dictionary and looked up the cache through `__dict__` for every decoded token, even on hits. `_encode_plain()` used the same pattern per segment.

Cache contents and limits, BPE merge order, special-token handling and byte decoding are unchanged. The two empty dictionaries are allocated at construction rather than first use.

This PR contains only:
- `tools/strata_tokenizer.py`
- `tools/test_strata_tokenizer.py`

No parser optimizations, profiling infrastructure, benchmark files, or native/GPU changes are included.

## Isolated validation

Candidate and untouched baseline both started at upstream `fb58e0dbc8399662c0e47c76578c6e878b14f6cf`. Server, frontend and benchmark-fixture source hashes were identical between checkouts; only the tokenizer and its test file differed.

```text
python -m unittest tools.test_strata_tokenizer serve.test_detok
Ran 15 tests — OK (skipped=5)
```

**10 tests passed; five tests requiring unavailable tokenizer assets were skipped.** Added checks cover:
- Decoding before any encoding, cold and warm, including Unicode/CRLF/NUL and all 256 raw byte tokens.
- Invalid IDs before and after cache use.
- Separate tokenizer instances with different vocabularies, exercising both piece and byte cache independence.
- Concurrent first-use decoding on a shared tokenizer.

Existing tests cover cold/warm/disabled piece caching, cache bounds, synthetic BPE parity and concurrent encoding. `git diff --check` passed.

**Windows correctness and performance validation: pending.**

## Same-day CPU measurements

Intel Core i7-12700KF, Linux 7.0.0-38-generic, glibc 2.39, CPython 3.12.3, regex 2026.9.29, Jinja2 3.1.6. Default scheduling/power policy, no affinity pinning. Seven fresh-process samples per checkout, alternating baseline/candidate order, two warmups per case, profiling disabled.

Deterministic synthetic vocabulary (seed 268, 1,500 merges), 49,664 tokens / 66,048 UTF-8 bytes. Caches are warm; fixture construction, validation and hashing are outside timing. No model or GPU required.

| Workload | Untouched upstream median | PR median | Elapsed reduction |
|---|---:|---:|---:|
| Incremental detokenization | 12.86 ms | 9.85 ms | 23.4% |
| Detokenization + upstream parser + collection | 88.70 ms | 84.93 ms | 4.3% |
| Warm encoding | 3.94 ms | 3.94 ms | Approximately unchanged |

Detokenization ranges: 12.68–13.13 ms upstream, 9.77–10.16 ms candidate. All 42 measured outputs matched fixtures and had matching hashes across checkouts and samples. These are synthetic Python CPU measurements, **not inference-throughput claims**; constructor/startup performance was not benchmarked.

### Reproduction

With `regex` and `jinja2` installed, run this from the root of the baseline and PR checkouts. It uses the repository's existing GPU-free tokenizer fixture and prints one warmed sample of each workload. Repeat in fresh processes and alternate checkouts for comparisons.

```python
import time
from serve.server import Detokenizer
from serve.frontend import OutputParser
from serve.test_detok import synthetic_tokenizer

tok = synthetic_tokenizer(seed=268, n_merges=1500)
code = "def accumulate(values):\n    return sum(x * x for x in values)\n# café 日本語 🙂\n"
text = ("The result is correct. café 日本語 🙂\n" + code) * 512
ids = tok.encode(text, parse_special=False)
assert tok.decode(ids, errors="strict") == text

def detokenize():
    detok = Detokenizer(tok)
    return "".join(detok.push(token) for token in ids)

def pipeline():
    detok, parser, chunks = Detokenizer(tok), OutputParser(thinking=False), []
    for token in ids:
        chunks.extend(e.text for e in parser.feed(detok.push(token)) if e.kind == "content")
    chunks.extend(e.text for e in parser.finish() if e.kind == "content")
    return "".join(chunks)

for name, fn, expected in (
    ("encode", lambda: tok.encode(text, parse_special=False), ids),
    ("detokenize", detokenize, text),
    ("pipeline", pipeline, text),
):
    for _ in range(2):
        assert fn() == expected
    start = time.perf_counter_ns()
    result = fn()
    elapsed_ms = (time.perf_counter_ns() - start) / 1e6
    assert result == expected
    print(name, elapsed_ms)
```

Related on strata.com

Editorial links to help you install, pick models, or read release notes — not part of the upstream thread.