#5776 [Other] POST /tokenize: the prompt a request would read, without running it

open · @sergey-zinchenko · 0 comentários · No GitHub

Server & APIModels & quantsSecurity

Descrição

## Problem and result
A client that plans its prompt budget against this server had no way to count the prompt this server would actually read: `/v1/messages/count_tokens` counts only Anthropic's shape, and OpenAI clients had to run a request (or count on their own tokenizer, which disagrees with the template and the image tokens). Before: no endpoint. After: `POST /tokenize` in vLLM's shape — send a `prompt` (plain text) or a chat `messages` conversation and get back the token ids, their count and the context length, without running the model. A chat request is read exactly as `/v1/chat/completions` reads it (the same template, tools, thinking level), and its images are counted as `prepare()` counts them, so the count a client plans from here is the count its request would take.

## Related discussion
https://github.com/Niko1221/Strata/issues/6330

## Approach
The image expansion of `prepare()` (the block that replaces each image placeholder with the run of image tokens its picture takes) is moved into a `Service` method (`expand_image_tokens`, `_fetch_images`) so the endpoint counts the same prompt a request would read; `prepare()`'s behavior is unchanged. DeepSeek-V4.1's own vision tower stays in `prepare()`; the endpoint refuses to count such a prompt, with a message pointing at the chat routes.

The route sits in the same POST handler (auth as the `/v1/*` routes): a union of a `prompt` (completion) and a `messages` (chat) form, as vLLM's `vllm/entrypoints/serve/tokenize/` has one. Extra fields are tolerated (vLLM's `extra="allow"`); the request's `model` is accepted like on the other routes (one model serves every name — the client's deployment id). `continue_final_message` is a 400 (not supported). Sampling fields are ignored. Response: `{"count", "tokens", "max_model_len", "token_strs": null}`; 409/429/503 as on the other routes (the engine's FIFO, a starting engine).

## Validation
`python -m pytest serve/test_server.py` — all pass.

Tests that cover this change:
- `TokenizeEndpoint` (new) — both forms, the rendered prompt vs the bare text, image tokens checked against the real request's `usage.prompt_tokens`, tools, extra fields, 400s
- `ImageMarkers`, `VisionTempFiles`, `MaxTokens` — the `prepare()` behavior the refactor touches
- `serve/test_ds_pictures.py`, `serve/test_dsv41.py`, `serve/test_vision_tool_image.py`, `serve/test_exif_orientation.py`, `serve/test_context_overflow.py`, `serve/test_mcp_media.py`

**Not run:** a live GPU engine (no card on this box) — the endpoint's engine-facing path is exercised through the `MockEngine`/fake-vision harness only.

## Limits
- `continue_final_message` is not supported (400) — vLLM accepts it, but Strata's templates always end a request with the generation prompt.
- DeepSeek-V4.1's own vision tower (`DsPictures`) is not counted: 400 with a message pointing at the chat routes.
- `token_strs` is always `null` (the ids are returned; the texts are not).

No site

Links install, modelos, releases.