贡献 / #1823
#1823 tokenizer: a long message's ids remembered whole, on top of the piece cache
open · @xrip · 0 评论 · 去 GitHub 看
说明
## What `tools/strata_tokenizer.py`: on top of 0.1.41's piece cache, a long piece of text between special tokens (one chat message, 1,024+ characters) keeps its ids too. The ids are the same; only the work is skipped. At most 256 texts are kept, and the cache is cleared when full. ## Why An agent sends its whole conversation again every turn. The piece cache skips the BPE merges of repeated pieces, but the pre-tokenizer regex still runs over the whole conversation each turn. Text between special tokens is already encoded on its own, so a message seen before can return its ids directly. ## Measured 2.5 GHz Xeon E5-2670 v2, before the piece cache existed (fork based on 0.1.38): at a 190K-token conversation, encoding took 1.06 s per turn before and 0.042 s with this cache. The ids were identical on all 36 turns of a coding-agent session (34K -> 190K tokens). How much this adds on top of the piece cache alone was **not measured yet**: the piece cache already removes the merges, so the gain is the regex pass over the old messages. `tools.test_strata_tokenizer` and `serve.test_detok` pass. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
本站相关内容
相关页面的快捷入口。