Pull requests / #1942
#1942 serve: the image encoder can run on a second PC over ssh (#1404)
open · @TraceRecursion · 0 comentarios · En GitHub
Setup & installServer & APIAMD / HIPNVIDIA / CUDAModels & quantsDocumentationWindowsLinux
Descripción
Issue: Resolves #1404 ## Summary The image encoder is already a separate process behind a line protocol (`ENC <image> <out>`), and it is not tied to the server's machine, so it can run on a second PC over ssh. That takes its VRAM off the card running the model: 1383 MiB measured, and the engine's own startup line goes 3112 -> 3641 experts (+1.00 GiB of expert cache). 2.83 s per 1024-token picture against 15.27 s on this CPU (5.4x), and 1.2 s a picture slower than a local GPU encoder. It does not make text faster: long-context text is the same in all three modes, because +1 GiB against a ~47 GiB expert set is ~2% coverage. A picture-latency and VRAM trade. ## What changed - `docs/REMOTE_VISION.md` (new): the page asked for in #1404 - the exact command, the config line, what happens when the second PC is down, and the measurements with what they were measured on. - `README.md`: one line under "Pictures" pointing at it. - `docs/DETAILS.md`: the `"remote": true` paragraph in "Images (vision)", where the other vision settings are. - `tools/remote-vision/` (new): `strata_vision_remote.py` (the wrapper - the protocol passes paths, not data, so the picture and the embeddings cross one ssh connection), `strata-vision-remote.cmd` (the `vision.exe` shim; `Popen` cannot start a `.py` directly), `make_vocab_stub.py` (an 11 MB vocabulary-only stub - `strata-vision` opens the text model with `vocab_only = true`, so the vision host does not need the 51 GB shard), `remote-build.sh` (builds the encoder on a Linux host), `remote-vision-watchdog.sh` (reaps the encoder when the server's port stops answering), and `test-wrapper.ps1` (the check in the notes below). - `serve/server.py`: the two changes I asked about in #1404. `"remote": true` keeps the encode out of the request FIFO: a local encoder shares this machine's GPU with the engine and must stay serialised, a remote one shares nothing, and holding the FIFO across the upload to a starved host froze the whole API, text requests included. `Service.ensure_vision` lets the server start and serve text when the encoder cannot, answering `400 the image encoder is not available: ...` on the first image and asking again on the next one. Without it, a boot-order race dropped the connection and told the client nothing at all (measured: `status 0`). - `serve/test_server.py`: 4 tests (`RemoteVision`). - `.gitignore`: `tools/remote-vision/*.gguf`, so the generated stub cannot be committed by accident. ## Extra Notes **Default path.** `"remote": true` is the opt-in and nothing without it behaves differently - including the startup tolerance, which is gated on the same key: a LOCAL encoder that fails to start is still fatal, and still names the cause (#1445), because that is a broken install and not an absent host. Only a remote one starts text-only and retries. **What I checked.** `python -m unittest discover -s serve -t . -p "test_*.py"`: 694 tests, OK (11 skipped). `tools/remote-vision/test-wrapper.ps1` against the vision host: `READY 2560`, a 640x480 picture `OK 300` in 0.77 s. Only Python and `tools/` are touched, no C++/CUDA/HIP/SYCL file, so no backend rebuild is involved. The measurements are in the page with the hardware, the engine version and the date. **What I did not test.** The server side was only exercised on Windows; the wrapper's Linux side (`remote-build.sh`, the watchdog) was built and smoke-tested on the vision host but not against a Linux Strata server. `STRATA_VISION_MUX` is off by default because ssh multiplexing fails on Windows (`getsockname failed: Not a socket`). The translated READMEs are untouched. **The table is from engine 0.1.40.2** (the release current on 2026-10-07/08, when it was taken): the three modes were measured the same day on one build, which is what the deltas need. The encoder binary and its mtmd path are unchanged since, and the config gives `--vram-reserve-mib 700` explicitly, so 0.1.42's Windows auto-reserve floor (#1376) does not apply to these numbers either. I have not re-run the table on 0.1.42 - say the word and I will. **Left out on purpose:** my own deploy helper (a PowerShell script that pushes the sources to my vision host and rebuilds there - it hardcodes my two machines and my model paths). Happy to add a generalized version if you want one. **Happy to split** the `serve/server.py` behaviour and its tests into their own PR if that is easier to review; the page and `tools/remote-vision/` need each other to be usable.
En el sitio
Enlaces a install, modelos, releases.