#4307 [Other] serve+webui: unload that works — pre-staged checkpoints, polite waits, honest answers
open · @sakuumomo · 0 comentarios · En GitHub
Setup & installServer & APIAMD / HIPNVIDIA / CUDADocumentation
Descripción
Problem and result: Before: every Unload click paid for a synchronous save on the click — 36.7 s+ for a 4 GB conversation on the owner's desktop, with the web control appearing frozen; a click while busy answered 409 claiming "a request is running" even when the engine was actually saving; if the engine died on `/unload`, `/v1/unload` or `POST /load`, the server dropped the connection with a traceback. After: a background pass pre-stages checkpoints once the request queue has been quiet for one expected save length, so a quiet click costs teardown only (measured 2.4 s, `{"status": "unloaded"}`); a click during a save answers 409 in 0.6 ms with a body you can read: `a request is running or the model is saving; try again when it is quiet`; engine death answers 503 `{"error": {"type": "server_error"}}` on all three routes; and the last-active session is always parked on unload whatever its size (a 4.3 GB dirty park verified live, then restored as 152,129 tokens in a 3 s prompt read). All existing `status` fields keep their values, so current clients are unaffected.
Dependencies: stacked on #4306 — this branch is the re-based ladder branch (`..43a52ce9`, 16 commits) + 11 new commits (tip `110d1d7a`): review those eleven. Ten of the original twenty-one have landed in your tree since 0.2.12 (see the 2026-11-20 changes section). If #4306 merges first, this branch re-lands cleanly on `main`.
Related discussion: extends the same `/load` / `/unload` routes the chat-header PR #3550 uses; `docs/DETAILS.md` is updated in both PRs.
Approach: a daemon pre-save pass consults the ladder's gates before touching the request FIFO, saves with a non-blocking FIFO grab only when one is actually possible, and estimates the quiet window from the engine's own measured save durations scaled by conversation growth (size fallback, clamped 5–600 s); `session_ladder.pre_save_idle_s` pins or disables the estimate (`null` = adaptive, number = pinned seconds, 0 = any idle moment). A refused save backs off exponentially (5 s doubled, cap 300 s) instead of hammering a failing engine or disk. Retention order: an over-cap save retires an orphan or subagent (child) checkpoint first, then an aux-sized one, with plain least-recent as the fallback — a finished subagent's checkpoint must not evict the expensive main file. WebUI: the monitor page's Unload posts `?wait=240` (server waits politely for a gap, strictly under the browser's ~300 s fetch timeout) and keeps its label and state-pill consistent while the control request is in flight.
Validation:
Passed: full serve suite green on this branch tip and on the owner's rebuilt deploy branch (1386 passed, 17 skipped on the deploy branch rebuilt onto upstream `v0.2.8`; see Failed). Live on momo 2026-10-27: quiet unload 200 in 2.4 s with no save behind it; busy click 409 in 0.6 ms with the message quoted above; `?wait=240` click parked 4.3 GB of dirty conversation and unloaded in 53 s; same-session restores 144,590 / 152,129 tokens with 3 s prompt reads; a manual file deletion racing an in-flight save produced a logged `refused (io)`, the cooldown backed off, and the re-save landed clean 5 s later.
Failed: two `serve/test_lifeline.py` watcher tests fail identically on the unmodified base on this machine (environment-sensitive, pre-existing).
Not run: the post-engine-death backoff loop was not provoked live on the fleet box (pinned in the suite instead); moko renders no `session_ladder` block (build-gate verified only).
Limits: the adaptive gate deliberately never opens while requests chain (a busy agent loop is never disturbed — confirmed operationally); pre-stage writes cost up to the conversation size on disk under the 16-file retention cap.
## Changes since the last update (2026-11-07)
- Rebased onto upstream `v0.2.12` (`123bad4a`, engine 0.1.58); branch tip `b97e6edb` (still those 21 commits on top of the ladder branch). The same one request-path fusion as #4306: the token-seen accounting rides upstream's `seen = 0 if admission_rejected else prompt_tokens_seen(...)` shape.
- Gate refresh: full serve suite 1640 OK / 37 skipped on the deploy lineage carrying this branch; the pre-save pass, the 409 fast refusal, the 503 engine-death answers and the `?wait=240` WebUI contract are all pinned in that suite unchanged. Owner flavor builds (HIP gfx1030, CUDA+HIP hybrid gfx1150) green with installCheck on the redeployed train `nixos` @ `c3e4cdc4`. The runtime-requirements and title jobs are green on the new head; the PR is mergeable against current `main`.
Commits were re-signed (GPG ed25519) on 2026-11-07 after rebasing onto upstream `v0.2.12` (`123bad4a`); trees are unchanged from the originally validated work; branch tip `b97e6edb`; the pre-rebase tip `b764970e` stays fetchable as an ancestor of the train tag `pin-f6f00908`.
## Changes since the last update (2026-11-20)
- Rebased onto upstream `v0.2.17` (`12d491da4`); branch tip `110d1d7a` — 11 commits on the re-based #4306 ladder, down from twenty-one: your own #2497 shape of the `/load` 503 answer (`died_note`, split `EngineDied` / `EngineStuck`) landed upstream in 0.2.16 (`5205fc8bb`), so our older draft of that hunk is dropped from the branch, and the other dropped commits were absorbed in your tree the same way. What remains is the pre-save side: the idle-quiet pre-save pass (`pre_save`, adaptive quiet gate derived from measured save durations), the busy-409 wording that names the model saving, the bool-value refusal for numeric config keys, and the webui Unload wait. All `status` fields keep their values against your current tree.
- Gate refresh: the full serve suite runs 2209 tests with only the two environment-sensitive non-green entries (#4306's 2026-11-20 update); the pre-save pass, the 409 fast refusal, the 503 engine-death answers and the `?wait=240` WebUI contract are all pinned green in that suite.
- No CI checks were reported on the new head; the PR is mergeable against current `main`.
Commits were re-signed (GPG ed25519) on 2026-11-20 after the rebase; trees carry no changes beyond the rebase and the absorbed drops; the pre-rebase tip `b97e6edb` stays fetchable as an ancestor of the tag `pin-pre0217-pr-serve-unload-works`.
## Changes since the last update (2026-11-21)
Rebased onto the rebuilt ladder chain on upstream `v0.2.18` (`6e38f588a`); branch tip `110d1d7a` -> `4ce7c85e3`. 1:1 range-diff against the 0.2.17-era chain: the same eleven unload-specific commits, zero conflicts, zero adaptations. Gates re-run on the rebuilt lineage `nixos` @ `295872912` (88 signed commits): the full serve suite at 2249 tests with the same three upstream-`v0.2.18`/environment non-greens documented in #4306's 2026-11-21 update (none touches this chain's files), the load/unload-answering suites inside `test_server` 448 OK, and the engine-side batteries — CPU session family 7/7 with unchanged check counts, ThreadSanitizer green, HIP gfx1030 battery 20/20. The owner's desktop host was re-pinned to this lineage on 2026-11-21 and answers unload routes on it daily. Commits re-signed (GPG ed25519); trees unchanged from the validated work. The pre-rebase tip `110d1d7aa` stays fetchable as an ancestor of the tag `pin-4cfc06018`. No CI checks were reported on the new head; the PR is mergeable against current `main`.
En el sitio
Enlaces a install, modelos, releases.