#6452 [Engine] serve: an on-demand CPU picture reader starts without asking the expert cache for VRAM

open · @criskurtin · 0 comentarios · En GitHub

Setup & installServer & APIAMD / HIPModels & quantsDocumentationWindows

Descripción

## Problem and result

An on-demand ("elastic") picture reader that runs on the CPU by choice (no `"gpu"` in the run config's `vision` section) went through the GPU reader's start in `Service._vision_up`: the server first asked the engine's expert cache to keep the reader's `elastic_mib` of VRAM free. On an Arc Pro B390 with main 3142b66b9 (server and engine), the first picture logged:

```
[strata] images: the expert cache did not make room for the image encoder (VRAM needs an engine started with --vram-elastic (one NVIDIA GPU, --serve)); starting it anyway
[strata] images: the image encoder started in 1.0 s, 344 MiB of VRAM
```

The reader holds no VRAM. With this PR the same picture logs:

```
[strata] images: the image encoder started in 0.9 s, on the CPU
```

On an engine that can shrink its cache (`--vram-elastic`, one NVIDIA GPU), main's path would also keep the cache `elastic_mib` smaller until the reader's idle stop. That part is unconfirmed static analysis: the test below shows it with a fake engine, and I have no such GPU.

## Related discussion

none: a focused fix, found while testing #6448. Independent of the other PRs linked there.

## Approach

`Vision.cpu_only` is `not cfg.get("gpu")`. A CPU-only reader starts directly: no `engine.vram()` request, no `vision_held`, and the log line says "on the CPU". GPU readers, including one that falls back to the CPU, keep their path. `docs/VRAM_ELASTIC.md` lists the case.

## Validation

**Passed:**
- `terminal/serve/test_vision_elastic.py`: the new test starts an on-demand CPU reader against a shrinking and a non-shrinking fake engine (no VRAM request, nothing given back at the stop) and a GPU reader (it still asks). On main's `server.py` it fails with `[2048] != []` for both engines; here 24 passed.
- `uv run --python 3.12 --with-requirements terminal/requirements.txt --with-requirements terminal/requirements-dev.txt --with-requirements terminal/requirements-build.txt python tools/check --strict --only py-serve,py-tools,docs` at b02896323: py-serve 2056 passed, py-tools 1851 passed, docs 6 passed; 0 failed. `python terminal/tools/minimal_server_smoke.py --report smoke.json` with only `terminal/requirements.txt` installed: 63 tests, OK, 4 skipped.
- On the B390 (Fedora 44, kernel 7.2.11, compute runtime 26.35.39758), main 3142b66b9's SYCL engine, a run config with `"vision": {"gpu": false, "elastic": true, ...}` and `--vision`, served once from main's `terminal/` and once from this branch's, one chat request with a PNG data URI: the log lines above, and both servers read the picture's number correctly. Logs: https://github.com/xenonhr/Strata/tree/b390-evidence (`hw2-main.txt`, `server-reader-*-main.log`, `reader-*-main-answer.txt`).
- Fork CI at the same head, run by hand: [Config snapshot](https://github.com/xenonhr/Strata/actions/runs/44840590215), and every job of [Checks](https://github.com/xenonhr/Strata/actions/runs/44840585878) and [Minimal serving runtime](https://github.com/xenonhr/Strata/actions/runs/44840593575) but the tests under Failed.

**Failed:**
- In CI, two tests that fail the same way on main 3142b66b9 ([Checks](https://github.com/Niko1221/Strata/actions/runs/44699739037), [Minimal serving runtime](https://github.com/Niko1221/Strata/actions/runs/44699738802)): `terminal.tools.test_strata_mcp.Install::test_a_start_that_came_in_before_a_cancel_does_not_launch_the_job` on Windows, and `test_abort_sync` in Minimal serving runtime (`from serve import server`: no module named `serve`).

**Not run:**
- An engine with `--vram-elastic` (one NVIDIA GPU): the cache-shrinking case is shown with the test's fake engine only.
- Readers on AMD or Mac servers.

## Limits

On an engine without `--vram-elastic` (the B390's included) the change removes one VRAM request and the misleading note at the first start; later starts on main already say nothing.

Prepared with an AI coding assistant (Claude Code); every result above was observed on the machine named in Validation.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

En el sitio

Enlaces a install, modelos, releases.