Pull requests / #529

#529 serve: an image in an Anthropic tool_result reaches the encoder (Claude Code's Read of a picture)

closed · @sergqwer · 0 commentaires · Sur GitHub

Server & APINVIDIA / CUDAModels & quantsWindows

Description

Claude Code returns a picture it opens with `Read` (a screenshot, a diagram) as an image block **inside a `tool_result`**. `anthropic_to_messages` turned every `tool_result` into its text (`_text_of`), so the image never reached the encoder and the model described a picture it had not seen. A user's own pasted image (an image block at the top level of a user message) already worked; only the tool path lost it.

**Fix** (`serve/frontend.py`, `anthropic_to_messages`):
- a `tool_result`'s content goes through `_parts_of`, as the OpenAI path's `tool` messages already do; the chat template renders the image item inside `<tool_response>` (its `render_content` handles images in any role);
- an image block sent beside tool results in the same user turn is kept too (it used to be dropped by the same branch);
- a text-only tool result is still one string, so every text prompt renders exactly as before.

**Test:** `ImageMarkers.test_anthropic_tool_result_image`: an image alone, after text, beside a text tool result, and in both places; it counts the image tokens and checks where they land. It fails on main (0 image tokens instead of 3) and passes here. All serve tests: 159 OK (7 skipped), on 0.1.36.

**End to end** (Windows, RTX 5090, Flash-Next with the CPU encoder, Claude Code asked to `Read` a PNG with known text and shapes):
- before: the request carrying the tool result added 35 tokens; the answer was invented ("Hello Vision Test", a red circle);
- after: it added 391 tokens (the image's ~360); both texts with their colours and both shapes with their positions were right, and a dark-theme code screenshot was transcribed verbatim.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01VZy1yKaDDiA8a7svdwaHio

Sur le site

Liens install, modèles, releases.