Pull requests / #3336
#3336 [Other] Exclude refused batch admissions from completion metrics
open · @oscarlius · 0 Kommentare · Auf GitHub
Server & APIAMD / HIPNVIDIA / CUDAWindows
Beschreibung
## Problem and result
A request refused by every BGEN slot returns HTTP 503, but its history row is still (finish, prompt_tokens, prompt_read) = (length, 51, None), and successful completion/token/latency metrics include it. This fix records an error history row with zero processed prompt tokens and excludes this pre-admission rejection from successful completion and latency totals. The regression also checks Retry-After, Prometheus metrics, slot cleanup and a later successful request.
## Related discussion
Continues #3042 after the 0.2.0-beta tracker reset. The original passed the new format check, but GitHub refused to reopen the rebased branch; [the reset bot](https://github.com/Niko1221/Strata/pull/3042#issuecomment-6282593730) explicitly suggested a new linked pull request.
Closes #3040. This is an accounting follow-up to #1994; admission and retry policy stay the same.
## Approach
Classify GpuBusy as a pre-admission refusal only when no DONE segment has settled and no output token was emitted. Preserve accounting for settled or partly generated work.
## Validation
**Passed:**
- `py -3.12 -m ruff check --output-format json serve/server.py serve/test_parallel.py` (Ruff 0.2.1): 21 existing diagnostics, same as unmodified main; zero new diagnostics.
- Windows, Python 3.12.7: `python -m unittest serve.test_parallel.ParallelService.test_bgen_refused_for_good_is_a_retryable_503_and_frees_the_slots -v` — 1 passed on `23a4f428d78655b92b05048620570465469533b1`.
- `python -m unittest serve.test_parallel serve.test_prometheus -v` — 48 tests, passed; final test result `OK`.
- `python -m py_compile serve/server.py serve/test_parallel.py` and `git diff --check e88ee39d1d6be6e3364183af3b39b62a7ed6f211...HEAD` passed.
**Failed:**
- Expected red baseline: with exact upstream `serve/server.py` at `e88ee39d1d6be6e3364183af3b39b62a7ed6f211` (0.2.0-beta), the same focused test fails: `('length', 51, None) != ('error', 0, None)`. No candidate test failure observed.
**Not run:**
- Real model generation, CUDA/HIP/SYCL/Metal builds or hardware execution. This patch touches only Python server code and its CPU fake-engine tests.
## Limits
Fresh Python processes and loopback HTTP prove the server accounting behavior, not native engine admission or model quality. No runtime default is changed.
Mehr auf der Site
Links zu Install, Modellen, Releases.