Issues / #3171

#3171 [Bug]: elastic vision — a tool-returned image crashes the request (400 "malformed", AttributeError on a None encoder proc)

open · @Lithivm · 1 comentarios · En GitHub

Server & APINVIDIA / CUDAModels & quantsWindows

Descripción

## What happened

With the elastic image encoder (`vram_elastic: true` + vision `"elastic": true`), **any request that carries an image a tool returned fails with a 400** and an internal `AttributeError`, because the encoder is not running yet and nothing starts it on that path.

```
{"error": {"type": "invalid_request_error",
           "message": "the request is malformed (AttributeError: 'NoneType' object has no attribute 'stdin')"}}
```

An agent hits this continuously: a tool that reads an image file (Claude Code's `Read` of a screenshot, an MCP tool returning a plot) breaks the whole turn — and since clients resend history, the same picture can break the next turns too.

## Reproduction — no waiting involved

The encoder has not started in a fresh elastic session (it starts with the *first picture*), so this reproduces on the first request:

1. A model config with `"vram_elastic": true` and `"vision": {"gpu": true, "elastic": true, ...}`; start the server. `strata-vision.exe` is **not** running (expected for elastic).
2. Send one request whose messages contain an image in a **tool** message — Anthropic dialect, e.g.:

```json
POST /v1/messages
{"model": "...", "max_tokens": 120, "messages": [
  {"role": "user", "content": "read this"},
  {"role": "assistant", "content": [{"type": "tool_use", "id": "t1", "name": "Read", "input": {}}]},
  {"role": "user", "content": [{"type": "tool_result", "tool_use_id": "t1",
    "content": [{"type": "image", "source": {"type": "base64", "media_type": "image/png", "data": "<any png>"}}]}]}]}
```

3. **400** with the message above; the engine log shows nothing, the traceback only appears on the server's stdout.

The same request succeeds with `"elastic": false` (the encoder is started with the model, so `Vision.proc` is never `None`), which is the current workaround. A *user*-sent image works too — only the tool-returned path fails.

## Traceback

```
[strata] 400 malformed request (AttributeError: 'NoneType' object has no attribute 'stdin'):
Traceback (most recent call last):
  File "D:\Strata\serve\server.py", line 5508, in _post
    self._responses(req)
  File "D:\Strata\serve\server.py", line 5899, in _responses
    ids, thinking, tools, max_new, asm, validator, req = self._responses_prepare(req, cancel)
  File "D:\Strata\serve\server.py", line 6019, in _responses_prepare
    ids, thinking, max_new = svc.prepare(messages, shown, kw, req.get("max_output_tokens") or 0, req=req, ...)
  File "D:\Strata\serve\server.py", line 3854, in prepare
    fetched = self._note_unreadable_tool_images(messages, cancel)
  File "D:\Strata\serve\server.py", line 3840, in _note_unreadable_tool_images
    self.vision.encode(src)
  File "D:\Strata\serve\server.py", line 2426, in encode
    self.proc.stdin.write(f"ENC {img.name} {out.name}\n")      # relative to the encoder's cwd (#480)
AttributeError: 'NoneType' object has no attribute 'stdin'
```

(line numbers are 0.1.44.1; it arrives through `/v1/responses` here and through `/v1/messages` in the repro)

## Cause

The two places that encode a request's pictures are not symmetric. `prepare()` starts an elastic encoder before encoding:

```python
# prepare(), the request's own images
with self._turn(cancel):
    self._vision_up()                       # ← present
    encoded = self.vision.encode_all(images, cancel) ...
```

`_note_unreadable_tool_images()` (called earlier from `prepare()`, for pictures a tool returned) encodes without it:

```python
# _note_unreadable_tool_images()
with self._turn(cancel):                     # the encoder takes its turn with the requests (see below)
    self.vision.encode(src)                  # ← no _vision_up(): Vision.proc is None while an elastic encoder is down
```

`Vision.encode()` writes to `self.proc.stdin` unconditionally, so with an elastic encoder that has not started (or has stopped after `elastic_idle_s`) it raises `AttributeError`. That function's `except (ValueError, OSError)` does not catch it, so the exception escapes `prepare()` and lands in `_post`'s generic `except (TypeError, KeyError, AttributeError, IndexError, UnicodeError)`, which reports **any** such internal error as `400 invalid_request_error` ("the request is malformed") — hence a server bug that looks like a client bug. This is present on `main` (76e51103): `prepare()` has the `_vision_up()` call at line 4345-4346, `_note_unreadable_tool_images()` has none.

## Suggested fix

Start the encoder there too, exactly as the sibling path does (PR follows):

```diff
                          with self._turn(cancel):  # the encoder takes its turn with the requests (see below)
+                             self._vision_up()
                              self.vision.encode(src)
```

`_vision_up()` is a no-op when the encoder is not elastic or is already running, so this changes nothing for non-elastic setups.

Two smaller things seen on the way, both separate from the fix above:

- `_note_unreadable_tool_images()` exists so that an unreadable tool picture becomes `[image omitted: ...]` instead of a 400. Widening its `except` to `AttributeError` (or to `Exception`) would keep that promise for unexpected failures as well — right now an internal error on this path defeats the function's own purpose.
- `_post`'s generic handler maps **any** `AttributeError` raised while preparing a request to `400 invalid_request_error`. That is how this surfaced as "the request is malformed": the message sent the reporter looking at their own client. An internal error is a 500.

## Environment

- Strata 0.1.44.1, engine 0.1.44.1 (release zip), Windows 11 Pro 10.0.26200
- RTX 5070 Ti 16 GiB, driver 616.64; Ryzen 7 9700X, 31.6 GiB RAM
- Model Q2_0, context 131072, `--kv q4_0`, `--mmap-experts`, `--vram-reserve-mib 1600`, `--vram-elastic`
- `"vision": {"gpu": true, "max_tokens": 1024, "elastic": true, "elastic_mib": 3584, "elastic_idle_s": 120}`, `"vram_elastic": true`

Related but different: #2454 (an elastic encoder re-starts on every turn whose history re-sends a cached picture — a churn/wall-time issue, not a crash).

En el sitio

Enlaces a install, modelos, releases.