Pull requests / #1942

#1942 serve: the image encoder can run on a second PC over ssh (#1404)

open · @TraceRecursion · 0 comentarios · En GitHub

Setup & installServer & APIAMD / HIPNVIDIA / CUDAModels & quantsDocumentationWindowsLinux

Descripción

Issue: Resolves #1404

## Summary

The image encoder is already a separate process behind a line protocol (`ENC <image> <out>`), and it is not tied
to the server's machine, so it can run on a second PC over ssh. That takes its VRAM off the card running the
model: 1383 MiB measured, and the engine's own startup line goes 3112 -> 3641 experts (+1.00 GiB of expert
cache). 2.83 s per 1024-token picture against 15.27 s on this CPU (5.4x), and 1.2 s a picture slower than a
local GPU encoder. It does not make text faster: long-context text is the same in all three modes, because
+1 GiB against a ~47 GiB expert set is ~2% coverage. A picture-latency and VRAM trade.

## What changed

- `docs/REMOTE_VISION.md` (new): the page asked for in #1404 - the exact command, the config line, what happens
  when the second PC is down, and the measurements with what they were measured on.
- `README.md`: one line under "Pictures" pointing at it.
- `docs/DETAILS.md`: the `"remote": true` paragraph in "Images (vision)", where the other vision settings are.
- `tools/remote-vision/` (new): `strata_vision_remote.py` (the wrapper - the protocol passes paths, not data, so
  the picture and the embeddings cross one ssh connection), `strata-vision-remote.cmd` (the `vision.exe` shim;
  `Popen` cannot start a `.py` directly), `make_vocab_stub.py` (an 11 MB vocabulary-only stub - `strata-vision`
  opens the text model with `vocab_only = true`, so the vision host does not need the 51 GB shard),
  `remote-build.sh` (builds the encoder on a Linux host), `remote-vision-watchdog.sh` (reaps the encoder when the
  server's port stops answering), and `test-wrapper.ps1` (the check in the notes below).
- `serve/server.py`: the two changes I asked about in #1404. `"remote": true` keeps the encode out of the request
  FIFO: a local encoder shares this machine's GPU with the engine and must stay serialised, a remote one shares
  nothing, and holding the FIFO across the upload to a starved host froze the whole API, text requests included.
  `Service.ensure_vision` lets the server start and serve text when the encoder cannot, answering `400 the image
  encoder is not available: ...` on the first image and asking again on the next one. Without it, a boot-order
  race dropped the connection and told the client nothing at all (measured: `status 0`).
- `serve/test_server.py`: 4 tests (`RemoteVision`).
- `.gitignore`: `tools/remote-vision/*.gguf`, so the generated stub cannot be committed by accident.

## Extra Notes

**Default path.** `"remote": true` is the opt-in and nothing without it behaves differently - including the
startup tolerance, which is gated on the same key: a LOCAL encoder that fails to start is still fatal, and still
names the cause (#1445), because that is a broken install and not an absent host. Only a remote one starts
text-only and retries.

**What I checked.** `python -m unittest discover -s serve -t . -p "test_*.py"`: 694 tests, OK (11 skipped).
`tools/remote-vision/test-wrapper.ps1` against the vision host: `READY 2560`, a 640x480 picture `OK 300` in
0.77 s. Only Python and `tools/` are touched, no C++/CUDA/HIP/SYCL file, so no backend rebuild is involved. The
measurements are in the page with the hardware, the engine version and the date.

**What I did not test.** The server side was only exercised on Windows; the wrapper's Linux side
(`remote-build.sh`, the watchdog) was built and smoke-tested on the vision host but not against a Linux Strata
server. `STRATA_VISION_MUX` is off by default because ssh multiplexing fails on Windows (`getsockname failed:
Not a socket`). The translated READMEs are untouched.

**The table is from engine 0.1.40.2** (the release current on 2026-10-07/08, when it was taken): the three modes
were measured the same day on one build, which is what the deltas need. The encoder binary and its mtmd path are
unchanged since, and the config gives `--vram-reserve-mib 700` explicitly, so 0.1.42's Windows auto-reserve floor
(#1376) does not apply to these numbers either. I have not re-run the table on 0.1.42 - say the word and I will.

**Left out on purpose:** my own deploy helper (a PowerShell script that pushes the sources to my vision host and
rebuilds there - it hardcodes my two machines and my model paths). Happy to add a generalized version if you
want one.

**Happy to split** the `serve/server.py` behaviour and its tests into their own PR if that is easier to review;
the page and `tools/remote-vision/` need each other to be usable.

En el sitio

Enlaces a install, modelos, releases.