Pull requests / #2874
#2874 serve: strata_prefix marks a chat whose history has pictures
open · @stevie67 · 0 comentários · No GitHub
Server & APIAMD / HIPNVIDIA / CUDAModels & quantsDocumentationWindows
Descrição
Issue: none open for this. Related, not resolved by it: #2255 (pinned prefixes filling the conversation cache).
## Summary
`"strata_prefix"` was ignored for any request with an image in it (`strata_prefix ignored: the request has images`), so one photo anywhere in a chat's history turned it off until that message left the context. Pictures before the boundary now count with the tokens they take in the prompt, and the prefix is marked as for a text-only request.
The use case: a chat client sends `{"messages": K}` with K the index of its final user message. That message carries per-turn text (recalled memories, the clock) that the next turn replays without, so the turn checkpoint never matches again and only the system-prompt checkpoint was reused. With the pin, each turn resumes where the replayed history ends. A chat that once contained a photo lost this until now.
## What changed
- `serve/server.py`
- The loop in `prepare()` that turns each `<|vision_start|><|image_pad|>` into one pad per image token moved, unchanged, into `Service.expand_image_pads(ids, counts)`. `prepare()` calls it with the encoded counts and keeps its "the prompt and its images do not match" check.
- `resolve_prefix()` takes the request's image token counts (`image_counts`, in prompt order) instead of `has_images`. `head_ids()` expands the rendered head's pictures with the same helper, so the longest common start runs through them. The refusal for images is gone.
- New guard: a boundary inside a picture's tokens is ignored with `strata_prefix ... ignored: it ends inside a picture` (only `{"tokens": N}` can land there; a message boundary cannot).
- `serve/test_server.py`, class `SharedPrefix`: three tests with a fake encoder whose two pictures take 3 and 5 tokens, so a wrong count or order misses the boundary:
- `test_messages_resolve_past_a_picture_in_the_history`: a photo in the history; the prefix ends at the final message's `<|im_start|>` and holds all 3 of its tokens.
- `test_a_picture_after_the_prefix_stays_out_of_it`: pictures on both sides; 3 tokens before the boundary, 5 after.
- `test_a_token_boundary_inside_a_picture_is_ignored`: `{"tokens": N}` inside the picture is ignored and said in the log.
The first two failed before the change (ignored / `KeyError: 'tokens'`); the third fails with the guard removed (`{'tokens': 225} != {}`).
- `docs/DETAILS.md`: the pinned-prefix paragraph says pictures count now, names the chat-client use, and gives the measurement below.
## What it leaves alone
- A request without `strata_prefix` takes the same path as before: `prepare()` runs the moved loop with the same counts. `test_without_the_field_nothing_changes` and the `ImageMarkers` tests (#150, #2228) pass unchanged.
- A request with `strata_prefix` and no pictures: `image_counts` is `None`, and `head_ids()` returns what it did.
- The engine: no change. `pin=N` is one more cut in the same read loop as the turn and system-prompt checkpoints, which already handle pictures.
- Python only; no CUDA, HIP or SYCL file is touched.
## How it was checked
- `python -m unittest discover -s serve -t . -p "test_*.py"` on this branch (`477ac13c` + this change, macOS, Python 3.14): 851 tests, 850 pass, 11 skipped. Against `main` without the change: 848 tests, 847 pass. The one failure is the same in both, `test_keepawake.KeepAwakeTests.test_service_holds_for_the_whole_request_including_early_close`, and it fails on this Mac without the change too.
- Live, on the DGX Spark port (#2419, engine 0.1.44.1; the same server patch applied there), GB10, Unsloth UD-Q4_K_XL, 131,072 context, `--kv int8`, images on, 2026-10-17. A four-turn chat with a 640x400 PNG in its first message, `{"messages": K}` on every turn with history, per-turn text in the final user message:
| Turn | Prompt | Reused | Read | Time to read |
|---:|---:|---:|---:|---:|
| 1 | 5,333 | 0 | 5,333 | 7.9 s |
| 2 | 5,478 | 5,392 | 86 | 0.96 s |
| 3 | 5,503 | 5,395 | 108 | 1.20 s |
| 4 | 5,527 | 5,421 | 106 | 1.18 s |
No `ignored` line in the server window. Turns 3 and 4 resumed at the pin of the turn before and still answered about the picture correctly (its colours and its text), so the picture's tokens come back intact from the pinned checkpoint.
- The same client in a real chat (23-31k tokens) before the change, with a photo 27 messages back: every turn reused only the system prompt's checkpoint and read 4.8-7.0k tokens (4.7-7.5 s) before answering.
## Not tested
- Other chat templates than the Qwen3.8 one, and `/v1/messages` and `/v1/responses` requests with pictures (they reach `resolve_prefix` through the same `prepare()`).
- `{"message": I, "chars": N}` with pictures before message I (covered by the same expansion, no test of its own).
- Pictures given by URL.
- Windows, and any GPU other than the GB10.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
No site
Links install, modelos, releases.