改动 / #2435
#2435 serve: an elastic image encoder that cannot start is a 503; its idle stop is logged on a layer split too
open · @i-Vladimir-i · 0 评论 · 去 GitHub 看
Server & APIMulti-GPUNVIDIA / CUDAModels & quantsDocumentation
说明
Two of the three points @q8atnight reported in #2196 (2x RTX 3090, `--layer-split 24`) are bugs in the server half of the elastic image encoder, now in 0.1.44: - **The first picture's OOM dropped the connection.** On a layer split the engine turns `--vram-elastic` off, so the expert cache never makes room; the encoder then failed to start and the `RuntimeError` from `_vision_up()` escaped `prepare()`. It is now an `EngineDied` with "the image encoder failed to start: ...", which every route answers with a `503`, as the load path already does for a non-elastic encoder; the next picture tries again. - **The idle stop logged nothing when the cache had not made room.** The line printed only with `vision_held`; it now always says the encoder stopped, and adds the cache's part only when there was one. - **At start**, `"elastic": true` on a layer split says that the encoder's VRAM has to be kept free with `--vram-reserve-mib`, instead of letting the first picture find out. `docs/VRAM_ELASTIC.md` says the same. Tests: two new cases in `serve/test_vision_elastic.py` (both fail without the change); the file passes 9/9. The third point, a per-stage segment cache so a layer split can lend the encoder VRAM from one card, is an engine change (CUDA, `--vram-elastic` per stage) and is not in this PR; we have no NVIDIA card to test it on. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
本站相关内容
相关页面的快捷入口。