Pull requests / #819
#819 frontend: images inside Anthropic tool_result blocks reach the model
closed · @christopherrobertbrooks-tech · 0 评论 · 在 GitHub 查看
描述
## What happens
Claude Code (and other clients of the Anthropic Messages API) return a Read of an image file as an image block
*inside a `tool_result`*. Strata drops that image, so the model answers about a picture it never saw.
Measured on engine 0.1.39, Qwen3.8-Flash-Next Coder IQ1_M, vision on (`--vision gpu`), a Tesla V100-PCIE-32GB, through
`/v1/messages` with thinking disabled. Same screenshot of a dashboard, same question ("list the bar rows in the left
panel, top to bottom"):
| Where the image is | Answer |
|---|---|
| a plain user message | "Swap, Net, Disk, Data" (correct) |
| inside a `tool_result` (how Claude Code sends it) | "A, B, C, D, E" (made up) |
In a real coding-agent session this showed up as the model saying a correct screenshot "came back black", and in another
run as "the window looks right" written without having seen it.
## Why
In `serve/frontend.py`, `anthropic_to_messages()` sends a user message to the image path only when it contains no
`tool_result`, and the `tool_result` branch builds the tool message with `_text_of(block.get("content"))`, which keeps
text and drops image blocks.
## The change
After the tool message, the result's image parts follow as a user turn — the chat template shows images in user turns —
with a one-line text marker, "(image returned by the tool above)". Text-only tool results are unchanged.
With the change, the `tool_result` case answers "Swap, Net, Disk, Data", and in a real agent session the model read its
screenshot and described the window correctly.
## Tests
- New: `ClientShapes.test_anthropic_image_inside_a_tool_result_reaches_the_model` — fails on 0.1.39, passes with the change.
- `python serve/test_server.py`: all tests OK.
Not checked: the OpenAI path's `tool` messages with image content (might have the same gap; happy to look if useful).
Found while running Strata as the builder of a local coding agent on a V100. The change and this description were drafted
with an AI assistant (Claude), tested on our machine, and reviewed by me.
站内延伸阅读
链到安装、模型与版本说明,便于 SEO/GEO,非官方 issue 正文。