Pull requests / #819

#819 frontend: images inside Anthropic tool_result blocks reach the model

closed · @christopherrobertbrooks-tech · 0 comments · View on GitHub

Server & APIModels & quants

Description

## What happens

Claude Code (and other clients of the Anthropic Messages API) return a Read of an image file as an image block
*inside a `tool_result`*. Strata drops that image, so the model answers about a picture it never saw.

Measured on engine 0.1.39, Qwen3.8-Flash-Next Coder IQ1_M, vision on (`--vision gpu`), a Tesla V100-PCIE-32GB, through
`/v1/messages` with thinking disabled. Same screenshot of a dashboard, same question ("list the bar rows in the left
panel, top to bottom"):

| Where the image is | Answer |
|---|---|
| a plain user message | "Swap, Net, Disk, Data" (correct) |
| inside a `tool_result` (how Claude Code sends it) | "A, B, C, D, E" (made up) |

In a real coding-agent session this showed up as the model saying a correct screenshot "came back black", and in another
run as "the window looks right" written without having seen it.

## Why

In `serve/frontend.py`, `anthropic_to_messages()` sends a user message to the image path only when it contains no
`tool_result`, and the `tool_result` branch builds the tool message with `_text_of(block.get("content"))`, which keeps
text and drops image blocks.

## The change

After the tool message, the result's image parts follow as a user turn — the chat template shows images in user turns —
with a one-line text marker, "(image returned by the tool above)". Text-only tool results are unchanged.

With the change, the `tool_result` case answers "Swap, Net, Disk, Data", and in a real agent session the model read its
screenshot and described the window correctly.

## Tests

- New: `ClientShapes.test_anthropic_image_inside_a_tool_result_reaches_the_model` — fails on 0.1.39, passes with the change.
- `python serve/test_server.py`: all tests OK.

Not checked: the OpenAI path's `tool` messages with image content (might have the same gap; happy to look if useful).

Found while running Strata as the builder of a local coding agent on a V100. The change and this description were drafted
with an AI assistant (Claude), tested on our machine, and reviewed by me.

Related on strata.com

Editorial links to help you install, pick models, or read release notes — not part of the upstream thread.