Pull requests / #3042
#3042 serve: exclude refused batch admissions from completion metrics
open · @oscarlius · 0 comments · View on GitHub
Description
A request rejected by every BGEN slot returns HTTP 503 but is still recorded as a `length` completion, charging its full prompt and incrementing success-style request and latency metrics. Classify a `GpuBusy` with no settled DONE segment and no output tokens as an admission rejection: keep an error history row with zero processed prompt tokens, and leave completed-request totals, latency histograms, and last-generation timings unchanged. Extend the existing BUSYALL regression to check 503 with `Retry-After: 5`, error history, JSON and Prometheus success/prompt/latency metrics, slot cleanup, and correct accounting for a subsequent successful request. Admission/retry policy and accounting for already-settled or partially generated work are unchanged; this is a follow-up to the behavior established by #1994. Validation on Windows with Python 3.12.7: - Extended regression fails before the fix (`length`, 51 prompt tokens) and passes after it. - 58 CPU fake/mock tests passed across parallel serving, Prometheus, monitor, status handover, cancelled-read accounting, and quiet cancellation. - Additional HTTP/Prometheus and post-recovery assertions passed in the final focused run. - Python compilation and `git diff --check` passed; Ruff introduces no findings. - Independent review found no actionable issues. No native backend files changed, no real model validation was performed, and no live model service was restarted. Closes #3040.
Related on this site
Quick links to related pages.