问题 / #3040
#3040 [Bug]: refused batch admissions count as successful completions
open · @oscarlius · 0 评论 · 去 GitHub 看
说明
### What happened When every attempted BGEN slot refuses admission, the server correctly returns retryable HTTP 503, but `_run()` still records the rejected request as a normal `length` completion. It adds the full prompt to completed-request totals and adds an end-to-end latency observation, despite receiving no tokens or DONE segment for that request. The existing CPU fake-engine `BUSYALL` scenario in `serve/test_parallel.py` reproduces this with one successful request and one rejected request: - HTTP response for the rejected request: 503. - Rejected history row: `finish="length"`, `prompt_tokens=51`, `output_tokens=0`, `engine_generated=null`. - Completed-request total: 2, although only one request generated a reply. - `/metrics` in Prometheus format exports this total as `vllm:request_success_total` and includes the unprocessed prompt tokens. Expected: preserve the 503 and `Retry-After` contract, retain an error record, and exclude a request rejected before admission from completed-request token and latency accounting. Subsequent successful requests should still be admitted and counted normally. ### Strata version, GPU, OS Upstream main at `477ac13`, Windows, Python 3.12.7. Reproduced using the existing CPU fake batch engine; no GPU or model weights are required. ### Engine log The fixture repeatedly emits `ERR BGEN: no such free slot (--batch 2) or a bad max_new`. The server exhausts the existing bounded retries and returns 503, as designed by #1994. This report concerns the resulting accounting, not that admission or retry policy.
本站相关内容
相关页面的快捷入口。