Pull requests / #3042

#3042 serve: exclude refused batch admissions from completion metrics

open · @oscarlius · 0 comentários · No GitHub

Server & APIWindows

Descrição

A request rejected by every BGEN slot returns HTTP 503 but is still recorded as a `length` completion, charging its full prompt and incrementing success-style request and latency metrics. Classify a `GpuBusy` with no settled DONE segment and no output tokens as an admission rejection: keep an error history row with zero processed prompt tokens, and leave completed-request totals, latency histograms, and last-generation timings unchanged.

Extend the existing BUSYALL regression to check 503 with `Retry-After: 5`, error history, JSON and Prometheus success/prompt/latency metrics, slot cleanup, and correct accounting for a subsequent successful request. Admission/retry policy and accounting for already-settled or partially generated work are unchanged; this is a follow-up to the behavior established by #1994.

Validation on Windows with Python 3.12.7:

- Extended regression fails before the fix (`length`, 51 prompt tokens) and passes after it.
- 58 CPU fake/mock tests passed across parallel serving, Prometheus, monitor, status handover, cancelled-read accounting, and quiet cancellation.
- Additional HTTP/Prometheus and post-recovery assertions passed in the final focused run.
- Python compilation and `git diff --check` passed; Ruff introduces no findings.
- Independent review found no actionable issues. No native backend files changed, no real model validation was performed, and no live model service was restarted.

Closes #3040.

No site

Links install, modelos, releases.