Pull requests / #3174

#3174 serve: start an elastic image encoder before encoding a tool-returned image (#3171)

open · @Lithivm · 0 Kommentare · Auf GitHub

Server & APINVIDIA / CUDAModels & quantsWindows

Beschreibung

Fixes #3171.

## The bug

`_note_unreadable_tool_images()` encodes a picture that a **tool** returned (Claude Code's `Read` of an image file, an MCP tool returning a plot) with `self.vision.encode(src)` — but unlike `prepare()`'s own image path it never calls `self._vision_up()`:

```python
# prepare(), the request's own images — correct
with self._turn(cancel):
    self._vision_up()
    encoded = self.vision.encode_all(images, cancel) ...

# _note_unreadable_tool_images(), a picture a tool returned — no _vision_up()
with self._turn(cancel):  # the encoder takes its turn with the requests (see below)
    self.vision.encode(src)
```

With the elastic encoder (`vram_elastic: true` + vision `"elastic": true`) the encoder is not running between pictures, so `Vision.proc` is `None` and `encode()`'s `self.proc.stdin.write(...)` raises `AttributeError`. That function's `except (ValueError, OSError)` does not catch it, so it leaves `prepare()` and lands in `_post`'s generic handler, which reports any of those as a client error:

```
400 {"type": "invalid_request_error",
     "message": "the request is malformed (AttributeError: 'NoneType' object has no attribute 'stdin')"}
```

So an agent doing a normal tool call breaks the turn — and, because clients re-send history, possibly the turns after it as well.

## Repro (0.1.44.1, no waiting needed)

A fresh elastic session never starts the encoder (it starts with the first picture), so the first request with a tool image fails:

1. `"vram_elastic": true` and `"vision": {"gpu": true, "elastic": true, ...}`; start the server — `strata-vision` is not running (expected).
2. One request whose messages hold an image in a **tool** message (Anthropic dialect: `user` → `assistant tool_use` → `user tool_result` with an `image` block).
3. 400 as above. Measured: fails before this patch, answers normally after it, and the encoder is then started and stopped as usual (`VRAM 3584 MiB kept free: the expert cache N -> M of N slots` in the engine log).

`"elastic": false` does not hit it (the encoder runs from the start), and a *user*-sent image was always fine — only the tool-returned path is affected.

## The change

Two lines, the same call the sibling path makes. `_vision_up()` returns immediately when the encoder is not elastic or is already running (`if not getattr(v, "elastic", False) or v.alive(): return`), so nothing changes for non-elastic setups.

```diff
                         with self._turn(cancel):  # the encoder takes its turn with the requests (see below)
+                            self._vision_up()   # an elastic encoder is down between pictures: start it first, as
+                                                # prepare()'s own image path does (else encode() writes to a None proc)
                             self.vision.encode(src)
```

`\ No newline at end of file` — no; the file's LF endings are unchanged (2 lines added, 0 removed).

## Two things I did not change (see #3171)

- That function exists so an unreadable tool picture becomes `[image omitted: ...]` instead of a 400; widening its `except` to `AttributeError` would keep that promise for unexpected failures too. Kept out of this PR to keep it to the defect.
- `_post` maps any `AttributeError` raised while preparing a request to `400 invalid_request_error`. That is why this looked like a client bug; an internal error arguably deserves a 500.

## Verified

- Before: 400 with the exact message above (0.1.44.1, Windows 11 26200, RTX 5070 Ti, Q2_0, context 131072, `--mmap-experts`, `--vram-elastic`, `elastic_mib` 3584).
- After: the same request answers; regression checks pass on plain text (`/v1/chat/completions`), a user image via `/v1/chat/completions` and via `/v1/messages`, and the tool image that used to fail.

Mehr auf der Site

Links zu Install, Modellen, Releases.