Pull requests / #333
#333 serve: add lazy startup and reliable Windows model cleanup
closed · @KadoBOT · 0 comentários · No GitHub
BenchmarksServer & APIAMD / HIPWindows
Descrição
Strata can unload and reload an idle model, but the HTTP server still loads it during startup. On a shared GPU, that prevents keeping the API available without first occupying the GPU. This adds opt-in cold startup and consolidates native-process shutdown so Windows' stdin reader receives EOF before the server waits for exit. ### Behavior - `--lazy` or `"lazy_load": true` starts the text-only API with the model unloaded. The first generation uses the existing reload path, including `before_load` and the free-VRAM guard from #208. Eager startup remains the default; lazy startup with vision is explicitly rejected. - The configured model remains discoverable while unloaded. `/v1/status` exposes `loaded` and `auto_load`, `/props` reports the reload capability, and `/api/health` aliases `/health`. - Add JSON `POST /v1/load` and `/v1/unload` aliases accepting an optional model name. They enforce the existing key and browser-Origin checks, reject a different model with 404, and return 409 while work is active or queued. Existing `/load` and `/unload` routes remain available. Explicit loading refreshes the idle timer. - Native shutdown sends `QUIT`, closes stdin, waits for exit, and releases the pump/pipe/log handles. If forced shutdown still times out, retain process ownership and report failure instead of claiming it was unloaded. Preserve Windows kill-on-close job containment. This extends #208 rather than reimplementing its idle-unload policy. It contains no model downloads, machine-specific configuration, or hardware tuning. ### Validation - Native Windows: `python -m unittest discover -s serve -p "test_*.py" -v` — 86 tests, OK, 3 optional tests skipped. - Python compilation and `git diff --check` passed. - New HTTP tests cover discovery without loading, first-request loading, unload/reload, explicit loading and idle-timer refresh, model matching, queued/busy refusal, API keys, and Origin/content-type checks. - Process tests cover an entirely unloaded constructor, EOF before waiting, and retaining ownership after a shutdown timeout. The rebased branch was verified with mock generation/process fixtures; no native GPU benchmark was run for this PR.
No site
Links install, modelos, releases.