Pull requests / #1823

#1823 tokenizer: a long message's ids remembered whole, on top of the piece cache

open · @xrip · 0 comentários · No GitHub

Server & API

Descrição

## What

`tools/strata_tokenizer.py`: on top of 0.1.41's piece cache, a long piece of text between special tokens (one chat
message, 1,024+ characters) keeps its ids too. The ids are the same; only the work is skipped. At most 256 texts are
kept, and the cache is cleared when full.

## Why

An agent sends its whole conversation again every turn. The piece cache skips the BPE merges of repeated pieces, but
the pre-tokenizer regex still runs over the whole conversation each turn. Text between special tokens is already
encoded on its own, so a message seen before can return its ids directly.

## Measured

2.5 GHz Xeon E5-2670 v2, before the piece cache existed (fork based on 0.1.38): at a 190K-token conversation,
encoding took 1.06 s per turn before and 0.042 s with this cache. The ids were identical on all 36 turns of a
coding-agent session (34K -> 190K tokens).

How much this adds on top of the piece cache alone was **not measured yet**: the piece cache already removes the
merges, so the gain is the regex pass over the old messages.

`tools.test_strata_tokenizer` and `serve.test_detok` pass.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

No site

Links install, modelos, releases.