Pull requests / #3336
#3336 [Other] Exclude refused batch admissions from completion metrics
open · @oscarlius · 0 コメント · GitHub で見る
Server & APIAMD / HIPNVIDIA / CUDAWindows
本文
## Problem and result
A request refused by every BGEN slot returns HTTP 503, but its history row is still (finish, prompt_tokens, prompt_read) = (length, 51, None), and successful completion/token/latency metrics include it. This fix records an error history row with zero processed prompt tokens and excludes this pre-admission rejection from successful completion and latency totals. The regression also checks Retry-After, Prometheus metrics, slot cleanup and a later successful request.
## Related discussion
Continues #3042 after the 0.2.0-beta tracker reset. The original passed the new format check, but GitHub refused to reopen the rebased branch; [the reset bot](https://github.com/Niko1221/Strata/pull/3042#issuecomment-6282593730) explicitly suggested a new linked pull request.
Closes #3040. This is an accounting follow-up to #1994; admission and retry policy stay the same.
## Approach
Classify GpuBusy as a pre-admission refusal only when no DONE segment has settled and no output token was emitted. Preserve accounting for settled or partly generated work.
## Validation
**Passed:**
- `py -3.12 -m ruff check --output-format json serve/server.py serve/test_parallel.py` (Ruff 0.2.1): 21 existing diagnostics, same as unmodified main; zero new diagnostics.
- Windows, Python 3.12.7: `python -m unittest serve.test_parallel.ParallelService.test_bgen_refused_for_good_is_a_retryable_503_and_frees_the_slots -v` — 1 passed on `23a4f428d78655b92b05048620570465469533b1`.
- `python -m unittest serve.test_parallel serve.test_prometheus -v` — 48 tests, passed; final test result `OK`.
- `python -m py_compile serve/server.py serve/test_parallel.py` and `git diff --check e88ee39d1d6be6e3364183af3b39b62a7ed6f211...HEAD` passed.
**Failed:**
- Expected red baseline: with exact upstream `serve/server.py` at `e88ee39d1d6be6e3364183af3b39b62a7ed6f211` (0.2.0-beta), the same focused test fails: `('length', 51, None) != ('error', 0, None)`. No candidate test failure observed.
**Not run:**
- Real model generation, CUDA/HIP/SYCL/Metal builds or hardware execution. This patch touches only Python server code and its CPU fake-engine tests.
## Limits
Fresh Python processes and loopback HTTP prove the server accounting behavior, not native engine admission or model quality. No runtime default is changed.
関連リンク
インストール・モデル・リリースへのサイト内リンク。