Pull requests / #2435

#2435 serve: an elastic image encoder that cannot start is a 503; its idle stop is logged on a layer split too

open · @i-Vladimir-i · 0 comentários · No GitHub

Server & APIMulti-GPUNVIDIA / CUDAModels & quantsDocumentation

Descrição

Two of the three points @q8atnight reported in #2196 (2x RTX 3090, `--layer-split 24`) are bugs in the server half of
the elastic image encoder, now in 0.1.44:

- **The first picture's OOM dropped the connection.** On a layer split the engine turns `--vram-elastic` off, so the
  expert cache never makes room; the encoder then failed to start and the `RuntimeError` from `_vision_up()` escaped
  `prepare()`. It is now an `EngineDied` with "the image encoder failed to start: ...", which every route answers with
  a `503`, as the load path already does for a non-elastic encoder; the next picture tries again.
- **The idle stop logged nothing when the cache had not made room.** The line printed only with `vision_held`; it now
  always says the encoder stopped, and adds the cache's part only when there was one.
- **At start**, `"elastic": true` on a layer split says that the encoder's VRAM has to be kept free with
  `--vram-reserve-mib`, instead of letting the first picture find out. `docs/VRAM_ELASTIC.md` says the same.

Tests: two new cases in `serve/test_vision_elastic.py` (both fail without the change); the file passes 9/9.

The third point, a per-stage segment cache so a layer split can lend the encoder VRAM from one card, is an engine
change (CUDA, `--vram-elastic` per stage) and is not in this PR; we have no NVIDIA card to test it on.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

No site

Links install, modelos, releases.