Pull requests / #2455
#2455 serve: an all-cached image turn never wakes the elastic encoder
open · @jerry78424 · 0 comentários · No GitHub
Server & APIMulti-GPUAMD / HIPNVIDIA / CUDAModels & quantsWindows
Descrição
## Title Issue: Resolves #2454 ## Summary An elastic image encoder is woken (seconds of start, VRAM the expert cache gives up) on every turn that re-sends pictures it has already encoded - which is what most clients do on every turn. A turn whose images are all in the hash cache needs no encoder process at all; it now skips the start. ## What changed - `serve/server.py`: `Vision.key_of()` / `Vision.cached()` - a hash-cache lookup that answers with the process down (the cache and its .sve files survive an unload, as before). `prepare()` maps the turn's images through the cache first: when every one is a hit the turn skips the encode FIFO, `_vision_up()` and the encode. A miss - alone or alongside hits - keeps today's path byte-for-byte: FIFO, make-room, start, encode. - `serve/test_vision_cache_wake.py` (new, CPU-only, no engine): a re-sent picture does not wake a stopped encoder; a new picture alongside cached ones still wakes it; the cache lookup answers with the process down. ## Extra Notes - What it leaves alone: non-elastic vision (the branch is only entered when `vision.elastic`), the start/stop/idle lifecycle itself, the encode path, and every native file (no CUDA/HIP/SYCL build needed). - Measured on RTX 5090 32 GB, Windows 11 (26200), engine 0.1.44.1, nvfp4, `elastic_idle_s: 10` - the table and verbatim log lines are in #2454: a text-only turn with a re-sent picture went from a 2.3 s start + 2085 MiB taken from the expert cache to 722 ms with no start line; a re-sent picture turn: 1277 ms, no start; new pictures unchanged (3979-4213 ms, still start). - How I checked: `serve/test_vision_cache_wake.py` fails on the pre-change tree (a stopped encoder wakes: starts 2 != 1) and passes with the change; the whole serve suite passes on both the base tree + fix (348 tests, 3 skipped) and this branch (vision trio: 30, OK). - Not tested: AMD/HIP and layer-split machines (a layer split refuses elastic anyway; its start failure is #2435), and no native files are touched.
No site
Links install, modelos, releases.