#1852 serve: tokenise the prompt from the last shared prefix, not from the start

Description

This reopens #567 on the new main, as Niko1221 suggested after the history rewrite. It supersedes #567 and closes nothing.

Every request re-encoded the whole conversation before the engine saw it. `PromptEncoder` keeps the last prompts with their ids and resume points, and encodes only the text that follows the longest shared prefix. The ids are the ones a full encode gives, token for token.

## Changes against #567

Rebased on v0.1.42 (61b3fb5d). #931's literals (control-token text inside a message stays text) go through the incremental encoder: the shared prefix ends where the two prompts' plain spans first differ. blange48 merged #567 with #931 the same way on port-567, and stays credited as co-author. f346dc6b's piece cache is kept, and the incremental encoder sits on top of it. Upstream's `tools/test_strata_tokenizer.py` tests are all kept, and ours are added.

Two fixes of #567's own behaviour:

- Every rendered prompt starts with the template's header, a special-token boundary, so a prompt from another conversation counted as extending the kept one and replaced it. A subagent's request cost the main conversation its reuse on the next turn (8,276 characters reused, then 236). Now a prompt replaces an entry only when it reuses more than half of that entry's text. Otherwise it is a new entry, and the server keeps one per slot of the engine's conversation cache (`--conversation-cache-slots`, 4 without it). Token counts (`/v1/messages/count_tokens`) read the kept prompts without storing one.
- A prompt refused because it does not fit the context is no longer kept by the encoder. Other refusals keep it, since the client resends the same prompt.

SECURITY.md now says that the server keeps the text and ids of its last few prompts in memory, and that no endpoint shows them.

## Numbers and tests

Measured on v0.1.41 with llama.cpp's qwen35 vocabulary, `Service.encode_prompt` per turn, 12 measured turns per pass, in a 10-CPU Linux container on an Apple M5 Pro:

- 125K-token agent conversation resent with one more ~500-token turn: 42.5 ms per turn on v0.1.41, 11.0 ms with this PR (3 passes, spread between passes 0.5 ms). Final branch, one pass: 11.7 ms.
- 89K-token prompt resent identically: 29.5 ms on v0.1.41, 6.6 ms with this PR. Final branch: 6.6 ms.
- Ids identical to v0.1.41 on all 84 compared turns, and on every turn of the final-branch run.
- On a slower CPU the absolute times are higher: sergqwer measured 221 ms to 5 ms at 125K on #567, before the piece cache.

`python -m pytest serve tools/test_strata_tokenizer.py`: 598 passed, 18 skipped, 0 failed (v0.1.41 alone: 584 passed, 11 skipped). The 7 extra skips are the qwen35-vocabulary tests, which need `STRATA_TOKENIZER` or llama.cpp's vocab file. Fuzz: 2,400 random plain/literal-span cases, incremental against full, and 2,400 random message prompts through `Service.encode_prompt` with control-token and think text quoted. All equal.

Five commits on v0.1.41:

- c319b113 serve: the prompt is tokenised from the last shared prefix, not from the start (co-author blange48)
- b98660e9 serve: #567's server tests take their reference from a full encode_prompt
- 854564d5 tools: llama.cpp's qwen35 vectors are checked with the qwen35 vocabulary
- 8941f1f4 serve, tools: #567's tests and ByteTokenizer without repeated code
- 96e6139b serve: #567's kept prompts, one per conversation, none refused for the context