改动 / #3380
#3380 [Other] serve: an all-cached image turn never wakes the elastic encoder
open · @jerry78424 · 0 评论 · 去 GitHub 看
Server & APIMulti-GPUAMD / HIPNVIDIA / CUDAModels & quantsSecurityDocumentationWindows
说明
## Problem and result An elastic image encoder is woken (seconds of start, VRAM the expert cache gives up) on every turn that re-sends pictures it has already encoded - which is what most clients do on every turn. A turn whose images are all in the hash cache needs no encoder process at all. Measured on RTX 5090 32 GB, Windows 11 (26200), engine 0.1.44.1, nvfp4, `elastic_idle_s: 10` (table and verbatim log lines in #2454): before - a text-only turn with a re-sent picture cost a 2.3 s start + 2085 MiB taken from the expert cache; after - 722 ms with no start line. A re-sent-picture turn: 1277 ms, no start. New pictures unchanged (3979-4213 ms, still start). ## Related discussion Resolves #2454. Follows #2455 (same branch and author; closed by the 0.2.0-beta tracker reset, and GitHub would not reopen it after the rebase force-push). Independent of the #2641 successor (the tool-picture wake lands in this same method - textual overlap only; the two changes conflict in neither order, verified by cherry-pick). ## Approach - `serve/server.py`: `Vision.key_of()` / `Vision.cached()` - a hash-cache lookup that answers with the process down (the cache and its .sve files survive an unload, as before) and marks a hit most recently used, as `encode()` does on its hit path. `prepare()` maps the turn's images through the cache first, for an elastic encoder only: when every one is a hit the turn skips the encode FIFO, `_vision_up()` and the encode. A miss - alone or alongside hits - keeps today's path byte-for-byte: FIFO, make-room, start, encode. - `serve/test_vision_cache_wake.py` (new, CPU-only, no engine): a re-sent picture does not wake a stopped encoder; a new picture alongside cached ones still wakes it; the lookup answers with the process down and keeps the re-sent picture from aging out of the 64-entry cache; a non-elastic turn keeps today's path even when every picture is cached. - `docs/VRAM_ELASTIC.md`, `docs/DETAILS.md`: both halves of the elastic rule - a picture's turn starts the encoder, a turn of pictures the cache knows takes neither the turn nor a start. ## Validation **Passed:** - Every new behavior fails on a tree without the code that provides it. - `python -m unittest serve.test_vision_elastic serve.test_vision_busy serve.test_vision_cache_wake serve.test_images`: the vision four 36 OK - at `e88ee39d` (v0.2.0-beta-12), rebased 2026-10-20, Windows 11 26200, Python 3.12 venv, CPU-only tests. - Full serve suite: 791 OK (15 skipped). - With #2641 cherry-picked on top: the files stay 19 + 19 OK. **Failed:** none observed. **Not run:** AMD/HIP and layer-split machines (a layer split refuses elastic anyway; its start failure is #2435). No native files are touched, so no native builds are required. ## Limits - What it leaves alone: non-elastic vision, the start/stop/idle lifecycle itself, the encode path, and every native file (no CUDA/HIP/SYCL build needed). - The measured numbers are one machine (RTX 5090, nvfp4); the mechanism - no process start for an all-cached turn - is hardware-independent, but the saved seconds are not claimed for other machines.
本站相关内容
相关页面的快捷入口。