改动 / #3336

#3336 [Other] Exclude refused batch admissions from completion metrics

open · @oscarlius · 0 评论 · 去 GitHub 看

Server & APIAMD / HIPNVIDIA / CUDAWindows

说明

## Problem and result

A request refused by every BGEN slot returns HTTP 503, but its history row is still (finish, prompt_tokens, prompt_read) = (length, 51, None), and successful completion/token/latency metrics include it. This fix records an error history row with zero processed prompt tokens and excludes this pre-admission rejection from successful completion and latency totals. The regression also checks Retry-After, Prometheus metrics, slot cleanup and a later successful request.

## Related discussion

Continues #3042 after the 0.2.0-beta tracker reset. The original passed the new format check, but GitHub refused to reopen the rebased branch; [the reset bot](https://github.com/Niko1221/Strata/pull/3042#issuecomment-6282593730) explicitly suggested a new linked pull request.

Closes #3040. This is an accounting follow-up to #1994; admission and retry policy stay the same.

## Approach

Classify GpuBusy as a pre-admission refusal only when no DONE segment has settled and no output token was emitted. Preserve accounting for settled or partly generated work.

## Validation

**Passed:**
- `py -3.12 -m ruff check --output-format json serve/server.py serve/test_parallel.py` (Ruff 0.2.1): 21 existing diagnostics, same as unmodified main; zero new diagnostics.

- Windows, Python 3.12.7: `python -m unittest serve.test_parallel.ParallelService.test_bgen_refused_for_good_is_a_retryable_503_and_frees_the_slots -v` — 1 passed on `23a4f428d78655b92b05048620570465469533b1`.
- `python -m unittest serve.test_parallel serve.test_prometheus -v` — 48 tests, passed; final test result `OK`.
- `python -m py_compile serve/server.py serve/test_parallel.py` and `git diff --check e88ee39d1d6be6e3364183af3b39b62a7ed6f211...HEAD` passed.

**Failed:**
- Expected red baseline: with exact upstream `serve/server.py` at `e88ee39d1d6be6e3364183af3b39b62a7ed6f211` (0.2.0-beta), the same focused test fails: `('length', 51, None) != ('error', 0, None)`. No candidate test failure observed.

**Not run:**
- Real model generation, CUDA/HIP/SYCL/Metal builds or hardware execution. This patch touches only Python server code and its CPU fake-engine tests.

## Limits

Fresh Python processes and loopback HTTP prove the server accounting behavior, not native engine admission or model quality. No runtime default is changed.

本站相关内容

相关页面的快捷入口。