Pull requests / #2343

#2343 MCP: a picture a tool returns reaches the model as a picture (#2130)

open · @dsvi · 0 comments · View on GitHub

Server & APIAMD / HIPNVIDIA / CUDAModels & quantsWindowsLinux

Description

## Why

Closes #2130.

This is vibe coded, but tested to work. Now the AI can ask for an image to look at, through MCP.
Greatly improves ability to debug GUIs, where AI has to see for itself the relative alignment and sizes of various controls.

An MCP tool result that carries an `image` block reached the model as a sentence about a picture:

```
[image (image/png, 2,337,795 bytes): not shown to the model]
```

`serve/mcp.py`'s `result_text()` joined the content blocks into one string, so a server that reads image files, takes screenshots or renders charts could hand a picture back and the model never saw it. The model *can* ask for the image with an MCP call — that is what #2130 asks for — only the way back was missing.

The same gap bites harder through a client that runs the MCP server on its own side. Chatbox (and the rest built on the AI SDK) never touches Strata's MCP path, and for `/v1/chat/completions` the SDK JSON-stringifies a tool's `content` output, so the base64 of the picture lands inside the tool message. One photo of a chat, in a real Chatbox conversation against a 262144-token context:

```
request exceeds the context window: prompt (1491022 tokens) leaves no room to answer in the context (262144); requests are never truncated
```

That is the whole conversation dying on the turn after the picture. The base64 costs 0.73 tokens a character with this model's tokenizer (1.37 characters per token, measured with the pack tokenizer rather than estimated).

## What it does

**1. A picture on Strata's own MCP path — `serve/mcp.py`.** `result_text()` became `result_parts()`: the same text as before, plus — only when the result carried pictures — that same content in the prompt's own part shape (`{"type": "image", "source": "data:…"}`, what `serve/frontend.py`'s `_parts_of` and `chat_template.jinja` already consume). `McpHub.call` also returns `parts` and `images`, and `run_with_mcp` appends `r["parts"] or r["text"]`. Block order and the newlines between blocks survive, so the picture sits where the server put it.

- Caps per result: 8 pictures of 32 MiB each (`MAX_IMAGE_BYTES` is `serve/server.py`'s `IMAGE_URL_MAX`, the chat's own image limit). Past a cap the picture is named in the text, as it always was.
- `max_result_chars` caps **text** only; a picture's size is capped where it is taken.
- A `resource` block whose `blob` has an `image/*` mime counts as a picture (the same base64, another wrapper). `audio`, `resource_link` and error results stay text, and Strata still never opens a path an MCP server names.
- Without a vision encoder, or for a picture it cannot decode: `[image omitted: …]` in its place and the chat goes on — the existing `_note_unreadable_tool_images` path, not a new one.
- The web app shows the picture in the tool block and resends it as an image part next turn; the base64 is dropped from `localStorage` (it is megabytes).

**2. A picture a client hands back as text — `serve/server.py`.** `Service._lift_pictures_from_tool_text`, at the top of `prepare()`: a base64 run of 1024 characters or more in a **tool's** string content that decodes to a known image magic (PNG / JPEG / BMP / GIF / WebP magic bytes, not the client's label) is taken out of the text and given to the encoder as an image part, with `[image (image/png, 1,528,438 bytes)]` left in the text where it was. Same caps as one MCP result.

## Numbers

That exact payload, IQ3_S pack tokenizer, RTX 3090, Qwen3.8-Flash-Next + mmproj:

| | before | after |
| --- | ---: | ---: |
| the tool message | 2,038,152 characters of text | one picture |
| the prompt | 1,486,632 tokens → refused with 400 | 239 tokens of text + 1,013 image rows |
| the model | never sees the picture | sees the picture |

The 1,013 rows are that photo's own cost as a picture, measured on the running server (a request with it as a user image: 1,067 prompt tokens, 54 without). So the turn goes from 1,491,022 tokens to about 1,250 — and it answers.

Cost of the scan itself: **10.6 ms** for that 2 MB tool message, **3.1 ms** for a 2 MB tool message with no picture in it, **0.1 ms** for a 20 KB one.

Live end to end: the `img` MCP server's `read_image` on a 4320×2432 JPEG (the server resizes it to a 640×360 PNG, 455 KB) arrives through the hub as one picture, parts in the order `['text', 'text', 'image']`; a red 160×120 PNG sent as an Anthropic `tool_result` image block is answered "Red" at 142 input tokens.

## What it leaves alone

- A result with no pictures produces byte-identical prompt text: `parts` is `None` and the tool message is the plain string, as before. The literal-prompt tests in `serve/test_mcp.py` pass unchanged.
- A client that sends the picture as a real image part (an Anthropic `tool_result`, an OpenAI `image_url`) never needed this and is untouched.
- A user's own message is never read this way — only `role: "tool"` string content.
- `/v1/messages` and `/v1/responses` do not run MCP, so nothing there changes; `GET /mcp` is unchanged.
- No C++ is touched, so no CUDA / HIP / SYCL rebuild is needed.

## How I checked

- `serve/test_mcp.py` — 29 tests, three new: image blocks become pictures (order, the note, the data URL), `resource` blobs and both caps, and the character cap cutting the text but not the pictures. Two of them run the whole tool loop against a fake encoder and count `<|image_pad|>` in the prompt the engine actually got.
- `serve/test_tool_text_images.py` — new, 7 tests: the Chatbox shape, a bare `data:` URL in a tool's text, long base64 that is *not* a picture (left exactly as the client sent it), a user's text (left alone), the caps, no encoder (the base64 is dropped rather than kept), and the resend-the-whole-history-every-turn case.
- Rebased on current `main` (296b59d), the whole `serve/` suite: **796 tests, 1 failure, 11 skipped**. The branch adds 36 tests of its own (29 + 7), all passing. The one failure is `test_keepawake.test_service_holds_for_the_whole_request_including_early_close`, which fails the same way on plain `main` on this box (a Windows sleep-allowance test on Linux) — it is not from this change.

Related on strata.com

Editorial links to help you install, pick models, or read release notes — not part of the upstream thread.