Pull requests / #819

#819 frontend: images inside Anthropic tool_result blocks reach the model

closed · @christopherrobertbrooks-tech · 0 comentários · No GitHub

Server & APIModels & quants

Descrição

## What happens

Claude Code (and other clients of the Anthropic Messages API) return a Read of an image file as an image block
*inside a `tool_result`*. Strata drops that image, so the model answers about a picture it never saw.

Measured on engine 0.1.39, Qwen3.8-Flash-Next Coder IQ1_M, vision on (`--vision gpu`), a Tesla V100-PCIE-32GB, through
`/v1/messages` with thinking disabled. Same screenshot of a dashboard, same question ("list the bar rows in the left
panel, top to bottom"):

| Where the image is | Answer |
|---|---|
| a plain user message | "Swap, Net, Disk, Data" (correct) |
| inside a `tool_result` (how Claude Code sends it) | "A, B, C, D, E" (made up) |

In a real coding-agent session this showed up as the model saying a correct screenshot "came back black", and in another
run as "the window looks right" written without having seen it.

## Why

In `serve/frontend.py`, `anthropic_to_messages()` sends a user message to the image path only when it contains no
`tool_result`, and the `tool_result` branch builds the tool message with `_text_of(block.get("content"))`, which keeps
text and drops image blocks.

## The change

After the tool message, the result's image parts follow as a user turn — the chat template shows images in user turns —
with a one-line text marker, "(image returned by the tool above)". Text-only tool results are unchanged.

With the change, the `tool_result` case answers "Swap, Net, Disk, Data", and in a real agent session the model read its
screenshot and described the window correctly.

## Tests

- New: `ClientShapes.test_anthropic_image_inside_a_tool_result_reaches_the_model` — fails on 0.1.39, passes with the change.
- `python serve/test_server.py`: all tests OK.

Not checked: the OpenAI path's `tool` messages with image content (might have the same gap; happy to look if useful).

Found while running Strata as the builder of a local coding agent on a V100. The change and this description were drafted
with an AI assistant (Claude), tested on our machine, and reviewed by me.

No site

Links install, modelos, releases.