#6471 [Engine] serve: send STOP when closing a batch admission mid-prompt

open · @larryllr · 0 commentaires · Sur GitHub

Server & APIAMD / HIPNVIDIA / CUDAWindowsLinux

Description

## Problem and result

In `StrataEngine.generate_batched()`'s `finally:` block (`terminal/serve/server.py`), an early consumer exit during `phase == "solo"` sends `STOP` before draining to `DONE`, while an early exit during `phase == "admit"` (a `BGEN` admission) drains to `BADM` without ever sending `STOP`.

When a consumer closes the generator during a `BGEN` prompt read (for example `GeneratorExit` while writing a `PP` heartbeat before `_control()` polls `cancel`), the engine is never told to abort. Each `PP` line resets `_drain_control()`'s no-progress timer, so the engine reads the entire abandoned prompt while holding `ctl` and starving waiting requests, emits the admission token, copies the session into the slot (`BADM <slot> 1`), and only then gets `BSTOP`ped.

After this fix, leaving `phase == "admit"` before the admission token arrives (`len(out) == adm_from`) sends `STOP` for the request's engine generation (`_send_for(born, "STOP")`), so the engine aborts the read at the next chunk boundary (`DONE 0 ... cancel`, `BADM <slot> 0`) and releases `ctl` and the slot immediately. Closing after the admission token has already been emitted (`len(out) > adm_from`) stays unchanged so `BatchLegAccounting` and slot handoff behave identically.

## Related discussion

none (searched open and closed issues/PRs for `BADM`, `quarantine`, `#4451`, `#5657`, and `#1182`).

## Approach

- In `terminal/serve/server.py` (`StrataEngine.generate_batched` `finally:`):
  - When `phase == "admit"` and `len(out) == adm_from` (still reading the prompt, no admission token emitted yet), call `self._send_for(born, "STOP")` before `self._drain_control("BADM", ...)`.
  - Also use `self._send_for(born, "STOP")` in `phase == "solo"` so a restart between the generation check and the write cannot send `STOP` to the replacement engine.
  - All native backends (`engine/src/program/generate.cpp`, `engine/src/qwen35/q36_engine.cu`, `engine/sycl/src/program/generate.cpp`) handle `STOP` during `BGEN` by aborting the prompt read and printing `DONE 0 ... cancel` followed by `BADM <slot> 0`, and reset `stop_req` at the start of the next request.
- In `terminal/serve/test_stream_ownership.py` (`BatchOwnership.test_closing_an_admission_mid_prompt_sends_stop`):
  - Add a deterministic regression test that closes a `BGEN` admission on its first `PP` heartbeat while two background slots are active (without `DEAFSTOP` and without setting `cancel`) and asserts `["BGEN", "STOP"]` is sent, `finish == "cancel"` with `generated == 0`, and a following client completes cleanly with no foreign frames.

## Validation

**Passed:**
- Revision `82ece4e` (based on `origin/main` `3142b66`), Windows 11, Python 3.12 (`PYTHONPATH=terminal`):
  - Before fix (new test against unpatched `3142b66`): failed as expected with `AssertionError: 'STOP' not found in ['BGEN 2 500 ...', 'BSTOP 2']` after 2.222s (read the full abandoned prompt and activated slot 2).
  - `python -m unittest serve.test_parallel.BatchLegAccounting.test_close_at_admission_token_does_not_publish_partial_totals serve.test_stream_ownership.BatchOwnership.test_closing_an_admission_mid_prompt_sends_stop -v`: `Ran 2 tests in 2.338s — OK`.
  - `python -m unittest serve.test_parallel serve.test_parallel_yield serve.test_admit_order serve.test_control_notify serve.test_stream_ownership serve.test_restart_waiters serve.test_quiet_cancel -v`: `Ran 122 tests in 391.382s — OK (skipped=1)` (1 skipped: Windows lacks `signal.SIGSTOP` for `test_an_undrainable_replica_is_not_handed_to_the_next_request`).
  - `git diff --check`: clean (exit 0).

**Failed:**
- none observed.

**Not run:**
- Native engine/GPU builds (CUDA, HIP, SYCL, Metal) and Linux-only `SIGSTOP` test: Python `terminal/serve/` change only; Linux CI checks (`check.yml`, `terminal-minimal-server.yml`) cover POSIX execution.

## Limits

This covers early consumer exits during `phase == "admit"` before the admission token arrives (`len(out) == adm_from`). No native engine C++ code was modified.

Sur le site

Liens install, modèles, releases.