#1997 before_load failure/busy-refusal is ignored ("loading anyway") - two engines co-load, one dies (exit code 1)

Beschreibung

**Summary:** Two Strata servers sharing one GPU via reciprocal `before_load` unload hooks: `/unload` answers `{"status":"busy"}` mid-turn instead of preempting, but `ensure_loaded` ignores the hook's exit code and loads anyway — both engines co-load onto a full card and one dies (exit code 1).

**Setup.** One RTX 5090 (32 GB), two Strata servers, each a 24 GiB quant (coder `:8080`, chat `:8081`). VRAM-exclusive by design: each server's `before_load` runs `POST /unload` against the peer before loading.

**What happens.** A request arrives for the idle lane while the busy lane is mid-turn. The hook fires `/unload`, which politely answers `{"status":"busy"}` — refusal, not preemption. `ensure_loaded` then logs `loading anyway` (return code of the hook is not honored) and loads the second 24 GiB quant onto a card that still holds the first. The engine dies with exit code 1 and the server restarts it "a minute or two" later. In a 15-minute replay of realistic concurrent traffic, **17/65 consumer requests got HTTP 503** and the coder engine crash-looped 4x.

Log signature (verbatim, engine log):

```
[strata] before loading: curl -s -m 5 -X POST -H "Content-Type: application/json" http://<peer>/unload
{"status": "busy"}[strata] loading the model again (it was unloaded) ...
[strata] the engine had stopped (exit code 1); starting it again (a minute or two) ...
```

**Workaround we shipped.** The hook polls `/unload` until the peer reports truly `unloaded`/`not loaded`. Two structural limits remain:
1. `before_load` runs under a hardcoded ~120 s subprocess timeout, so the poll budget is ~100 s — a 190 s peer turn outlives any sane wait, and when the poll gives up the load attempt fails as a *graceful* `min_free_vram` 503 (better than the crash, but still a consumer-visible cliff that scales with turn length).
2. The advisory-`busy` + ignored-exit-code combination means the **default hook shape everyone will write** (single `curl /unload`) produces silent co-load crashes. The dangerous failure is the one that looks successful.

**Asks.**
1. Honor `before_load` failure: if the hook exits nonzero (or the refusal is still in effect when `ensure_loaded` proceeds), **refuse the request with 503 + `Retry-After` instead of loading anyway.** A temporary refusal beats a wedged engine; today the inverse is true.
2. A blocking unload variant (`POST /unload` with `?wait=true`, or an `accepted/busy` progress contract) so orchestration can wait on the server's terms instead of racing a hardcoded subprocess timeout.
3. Papercut: `POST /unload` with an empty body and no `Content-Type` header returns **415** — a success-shaped endpoint that silently fails the most common one-liner call; an empty-body POST arguably deserves none, or the docs should call out the header.

Happy to contribute the contention-soak replay harness as a regression test. Related context: #1519 reports the same `exit code 1` symptom in a single-server setup (unverified overlap, but our data shows *any* VRAM squeeze at load time ends in the same crash); #1853/#1034 show the orchestration patterns users are converging on. Measured swap numbers if useful: first cold swap 29.8–30.6 s, preempt-mid-28k-prefill 50.7 s, warm swap-back 13.4 s.