#6474 [Engine] serve: keep a restarted engine's batch slot state out of old request cleanup

open · @larryllr · 0 评论 · 去 GitHub 看

Server & APIAMD / HIPNVIDIA / CUDAWindowsLinux

说明

## Problem and result

Two cleanup paths in `terminal/serve/server.py` still share the restart generation race fixed in `_release_slot_when_done()` (#4451 / #5657):

1. In `StrataEngine.generate_batched()`'s `finally:` block, `same = self.gen == born` is evaluated once at the top, before blocking calls (`_drain_control()` up to `DRAIN_TIMEOUT_S`, `_bdone_after_eos()` up to 2 s). If the engine dies and `restart()` advances `self.gen` and allocates new `slot_busy`, `slot_claims`, `slot_held`, `slot_used`, and `slot_live` tables during that wait, the stale `same == True` causes the old request's cleanup to clear `self.slot_live[slot]`, `self.slot_held[slot]`, `self.slot_busy[slot] = False`, and `self.slot_claims.discard(slot)` (and the same for `reserved`) on the **new** engine's active slots — freeing a slot owned by a live request (or raising `IndexError` if the restarted engine has fewer batch slots).
2. In `StrataEngine._quarantine_control()`, `self.slot_releasing`, `self.slot_claims`, and `self.slot_busy` are mutated without capturing the old generation's slot tables before checking `self.gen == born` (and inside `run()`, `self.gen == born` is checked before `self.alive()` and before acquiring `slot_cv`), so a restart right before entry or during quarantine can mutate the new engine's slot state.

After this fix, both paths follow `_release_slot_when_done()`'s invariant (`restart()` advances `self.gen` before replacing the per-process slot objects): they capture the slot lists/sets first, then check `self.gen == born` (after any blocking drains/waits) and bounds-check the slot index, mutating only the captured old-generation objects.

## Related discussion

Follow-up to the restart slot-table capture pattern in #4451 and #5657 (whose limits noted remaining restart/field-capture timing windows). Searched open and closed issues/PRs for `#5657`, `#4451`, `quarantine`, and `BADM`; none covers these two paths.

## Approach

- In `terminal/serve/server.py`:
  - `StrataEngine._quarantine_control()`: capture `busy, releasing, claims = self.slot_busy, self.slot_releasing, self.slot_claims` before checking `if self.gen != born or slot >= len(busy): slot = None`, and mutate those captured objects both on entry and inside `run()`.
  - `StrataEngine.generate_batched()` `finally:`: check `self.gen == born` freshly before `_bdone_after_eos()`, then after all drains/waits capture `busy, claims, held, used, live_slots` before evaluating `same = self.gen == born` and bounds-checking `reserved`/`slot` against `len(busy)`.
- In `terminal/serve/test_parallel_yield.py` (`ParallelServiceYield`):
  - Add two deterministic, sleep-free regression tests modeled on `test_a_restart_during_the_release_keeps_the_new_engines_claim`:
    - `test_a_restart_during_admit_drain_keeps_the_new_engines_slot`
    - `test_a_restart_during_quarantine_keeps_the_new_engines_slot` (covers both pre-entry restart and mid-quarantine restart after `self.gen == born` is checked).

## Validation

**Passed:**
- Revision `f18f89a` (based on `origin/main` `3142b66`), Windows 11, Python 3.12 (`PYTHONPATH=terminal`):
  - Before fix (both new tests against unpatched `3142b66`): failed as expected (`Ran 2 tests in 1.612s — FAILED (failures=2)` with `AssertionError: False is not true : the old request's finally freed the new engine's active slot` and `AssertionError: 0 not found in set() : pre-entry restart: quarantine erased the new engine's claim`).
  - `python -m unittest serve.test_parallel_yield.ParallelServiceYield.test_a_restart_during_admit_drain_keeps_the_new_engines_slot serve.test_parallel_yield.ParallelServiceYield.test_a_restart_during_quarantine_keeps_the_new_engines_slot -v`: `Ran 2 tests in 1.644s — OK`.
  - `python -m unittest serve.test_parallel serve.test_parallel_yield serve.test_admit_order serve.test_control_notify serve.test_stream_ownership serve.test_restart_waiters serve.test_quiet_cancel -v`: `Ran 123 tests in 393.662s — OK (skipped=1)` (1 skipped: Windows lacks `signal.SIGSTOP` for `test_an_undrainable_replica_is_not_handed_to_the_next_request`).
  - `git diff --check`: clean (exit 0).

**Failed:**
- none observed.

**Not run:**
- Native engine/GPU builds (CUDA, HIP, SYCL, Metal) and Linux-only `SIGSTOP` test: Python `terminal/serve/` change only; Linux CI checks (`check.yml`, `terminal-minimal-server.yml`) cover POSIX execution.

## Limits

This addresses restart generation isolation in `generate_batched()`'s `finally:` cleanup and `_quarantine_control()`. No native engine C++ code was modified.

本站相关内容

相关页面的快捷入口。