Issues / #2454

#2454 [Bug]: elastic image encoder wakes on every turn when the client re-sends pictures it already encoded

open · @jerry78424 · 0 commentaires · Sur GitHub

Setup & installServer & APIMulti-GPUAMD / HIPNVIDIA / CUDAModels & quantsWindows

Description

## What happened

With the elastic image encoder (`vram_elastic: true` + vision `"elastic": true`), every turn whose messages contain an image takes the encoder's start - even when every one of those images is already in the hash cache. `Vision.unload()` keeps `Vision.cache` and the `.sve` files (that is the documented design), so an already-encoded picture needs no encoder process at all - but `prepare()` never asks the cache: on any image turn it takes the encode FIFO and calls `_vision_up()` unconditionally, and only afterwards does `encode_all` find that everything is a hit.

This is not a corner case: most clients re-send the whole conversation - pictures included - on every turn. So after the first idle stop, **every subsequent turn pays the start again, forever**, including pure-text follow-ups.

Repro (any elastic setup, no rebuild):
1. Start a model with `vram_elastic: true` and vision `{"elastic": true, "elastic_idle_s": 10}`.
2. Send a message with an image -> encoder starts, answer arrives.
3. Wait past `elastic_idle_s` (log: `the image encoder stopped after 10 s without a picture`).
4. Send a text-only message while the client re-sends the picture from step 2.

Actual: the log shows `the expert cache keeps ... free` + `the image encoder started ...` before the answer - and the same on every later turn.
Expected: a turn whose images are all cached answers from the cache without waking the encoder. A new picture (alone or alongside cached ones) should still wake it.

Measured on RTX 5090 32 GB, Windows 11 (build 26200), engine 0.1.44.1, nvfp4 quant, `elastic_idle_s: 10`, a client that re-sends history images every turn. Before the change, one chat session's engine log (verbatim, one session file):

```
[strata] images: the image encoder started in 5.8 s, 2085 MiB of VRAM
[strata] images: the image encoder stopped after 10 s without a picture; the expert cache took its VRAM back (22528 of 22909 MiB)
[strata] images: the image encoder started in 2.3 s, 2085 MiB of VRAM
[strata] images: the image encoder stopped after 10 s without a picture; ...
[strata] images: the image encoder started in 2.3 s, 2085 MiB of VRAM
```

Per-turn wall time with a scripted client, one conversation, the same machine - before the change every image-bearing turn after an idle stop was preceded by such a start line in the engine log; with the fix:

| turn | before | with the fix |
|---|---|---|
| new picture A | 4213 ms, encoder starts | unchanged: starts (3979-4213 ms) |
| text-only, picture re-sent in history | a 2.3 s start line before the answer, 2085 MiB taken from the expert cache | 722 ms, no start line, no VRAM change |
| same picture re-sent again | another start line | 1277 ms, no start line |
| new picture B | starts | starts (3979 ms) |

The `2.3 s` and `2085 MiB` are the steady-state re-start cost on this card (5.1-5.8 s on the very first start).

## Strata version, GPU, OS

0.1.44.1 · RTX 5090 32 GB · Windows 11 (build 26200)

## Engine log

Excerpt above is verbatim from `strata-nvfp4-nvfp4.log`.

## Fix

serve/server.py only, no native files: `Vision.key_of()` / `Vision.cached()` (hash lookup that works with the process down), and `prepare()` maps the turn's images through the cache first - when every one is a hit the turn skips the FIFO, `_vision_up()` and the encode entirely. A miss keeps today's path exactly.

`serve/test_vision_cache_wake.py` (new, CPU-only, no engine): a re-sent picture does not wake a stopped encoder, a new picture alongside cached ones still wakes it, and the cache lookup answers with the process down. The whole `serve` suite passes with the change here (348 tests, 3 skipped).

What I did not test: AMD/HIP and layer-split machines (on a layer split `--vram-elastic` is refused anyway, and the start there is covered by #2435), non-elastic vision (path untouched), and the CUDA/HIP/SYCL builds (no native file changes).

I'll open a PR with the fix + the test referencing this issue.

Sur le site

Liens install, modèles, releases.