Pull requests / #3380

#3380 [Other] serve: an all-cached image turn never wakes the elastic encoder

open · @jerry78424 · 0 comments · View on GitHub

Server & APIMulti-GPUAMD / HIPNVIDIA / CUDAModels & quantsSecurityDocumentationWindows

Description

## Problem and result
An elastic image encoder is woken (seconds of start, VRAM the expert cache gives up) on every turn that re-sends pictures it has already encoded - which is what most clients do on every turn. A turn whose images are all in the hash cache needs no encoder process at all.

Measured on RTX 5090 32 GB, Windows 11 (26200), engine 0.1.44.1, nvfp4, `elastic_idle_s: 10` (table and verbatim log lines in #2454): before - a text-only turn with a re-sent picture cost a 2.3 s start + 2085 MiB taken from the expert cache; after - 722 ms with no start line. A re-sent-picture turn: 1277 ms, no start. New pictures unchanged (3979-4213 ms, still start).

## Related discussion
Resolves #2454. Follows #2455 (same branch and author; closed by the 0.2.0-beta tracker reset, and GitHub would not reopen it after the rebase force-push). Independent of the #2641 successor (the tool-picture wake lands in this same method - textual overlap only; the two changes conflict in neither order, verified by cherry-pick).

## Approach
- `serve/server.py`: `Vision.key_of()` / `Vision.cached()` - a hash-cache lookup that answers with the process down (the cache and its .sve files survive an unload, as before) and marks a hit most recently used, as `encode()` does on its hit path. `prepare()` maps the turn's images through the cache first, for an elastic encoder only: when every one is a hit the turn skips the encode FIFO, `_vision_up()` and the encode. A miss - alone or alongside hits - keeps today's path byte-for-byte: FIFO, make-room, start, encode.
- `serve/test_vision_cache_wake.py` (new, CPU-only, no engine): a re-sent picture does not wake a stopped encoder; a new picture alongside cached ones still wakes it; the lookup answers with the process down and keeps the re-sent picture from aging out of the 64-entry cache; a non-elastic turn keeps today's path even when every picture is cached.
- `docs/VRAM_ELASTIC.md`, `docs/DETAILS.md`: both halves of the elastic rule - a picture's turn starts the encoder, a turn of pictures the cache knows takes neither the turn nor a start.

## Validation

**Passed:**
- Every new behavior fails on a tree without the code that provides it.
- `python -m unittest serve.test_vision_elastic serve.test_vision_busy serve.test_vision_cache_wake serve.test_images`: the vision four 36 OK - at `e88ee39d` (v0.2.0-beta-12), rebased 2026-10-20, Windows 11 26200, Python 3.12 venv, CPU-only tests.
- Full serve suite: 791 OK (15 skipped).
- With #2641 cherry-picked on top: the files stay 19 + 19 OK.

**Failed:** none observed.

**Not run:** AMD/HIP and layer-split machines (a layer split refuses elastic anyway; its start failure is #2435). No native files are touched, so no native builds are required.

## Limits
- What it leaves alone: non-elastic vision, the start/stop/idle lifecycle itself, the encode path, and every native file (no CUDA/HIP/SYCL build needed).
- The measured numbers are one machine (RTX 5090, nvfp4); the mechanism - no process start for an all-cached turn - is hardware-independent, but the saved seconds are not claimed for other machines.

Related on this site

Quick links to related pages.