Pull requests / #3174
#3174 serve: start an elastic image encoder before encoding a tool-returned image (#3171)
open · @Lithivm · 0 comentarios · En GitHub
Server & APINVIDIA / CUDAModels & quantsWindows
Descripción
Fixes #3171.
## The bug
`_note_unreadable_tool_images()` encodes a picture that a **tool** returned (Claude Code's `Read` of an image file, an MCP tool returning a plot) with `self.vision.encode(src)` — but unlike `prepare()`'s own image path it never calls `self._vision_up()`:
```python
# prepare(), the request's own images — correct
with self._turn(cancel):
self._vision_up()
encoded = self.vision.encode_all(images, cancel) ...
# _note_unreadable_tool_images(), a picture a tool returned — no _vision_up()
with self._turn(cancel): # the encoder takes its turn with the requests (see below)
self.vision.encode(src)
```
With the elastic encoder (`vram_elastic: true` + vision `"elastic": true`) the encoder is not running between pictures, so `Vision.proc` is `None` and `encode()`'s `self.proc.stdin.write(...)` raises `AttributeError`. That function's `except (ValueError, OSError)` does not catch it, so it leaves `prepare()` and lands in `_post`'s generic handler, which reports any of those as a client error:
```
400 {"type": "invalid_request_error",
"message": "the request is malformed (AttributeError: 'NoneType' object has no attribute 'stdin')"}
```
So an agent doing a normal tool call breaks the turn — and, because clients re-send history, possibly the turns after it as well.
## Repro (0.1.44.1, no waiting needed)
A fresh elastic session never starts the encoder (it starts with the first picture), so the first request with a tool image fails:
1. `"vram_elastic": true` and `"vision": {"gpu": true, "elastic": true, ...}`; start the server — `strata-vision` is not running (expected).
2. One request whose messages hold an image in a **tool** message (Anthropic dialect: `user` → `assistant tool_use` → `user tool_result` with an `image` block).
3. 400 as above. Measured: fails before this patch, answers normally after it, and the encoder is then started and stopped as usual (`VRAM 3584 MiB kept free: the expert cache N -> M of N slots` in the engine log).
`"elastic": false` does not hit it (the encoder runs from the start), and a *user*-sent image was always fine — only the tool-returned path is affected.
## The change
Two lines, the same call the sibling path makes. `_vision_up()` returns immediately when the encoder is not elastic or is already running (`if not getattr(v, "elastic", False) or v.alive(): return`), so nothing changes for non-elastic setups.
```diff
with self._turn(cancel): # the encoder takes its turn with the requests (see below)
+ self._vision_up() # an elastic encoder is down between pictures: start it first, as
+ # prepare()'s own image path does (else encode() writes to a None proc)
self.vision.encode(src)
```
`\ No newline at end of file` — no; the file's LF endings are unchanged (2 lines added, 0 removed).
## Two things I did not change (see #3171)
- That function exists so an unreadable tool picture becomes `[image omitted: ...]` instead of a 400; widening its `except` to `AttributeError` would keep that promise for unexpected failures too. Kept out of this PR to keep it to the defect.
- `_post` maps any `AttributeError` raised while preparing a request to `400 invalid_request_error`. That is why this looked like a client bug; an internal error arguably deserves a 500.
## Verified
- Before: 400 with the exact message above (0.1.44.1, Windows 11 26200, RTX 5070 Ti, Q2_0, context 131072, `--mmap-experts`, `--vram-elastic`, `elastic_mib` 3584).
- After: the same request answers; regression checks pass on plain text (`/v1/chat/completions`), a user image via `/v1/chat/completions` and via `/v1/messages`, and the tool image that used to fail.
En el sitio
Enlaces a install, modelos, releases.