Issues / #3040

#3040 [Bug]: refused batch admissions count as successful completions

open · @oscarlius · 0 comentarios · En GitHub

Server & APIWindows

Descripción

### What happened

When every attempted BGEN slot refuses admission, the server correctly returns retryable HTTP 503, but `_run()` still records the rejected request as a normal `length` completion. It adds the full prompt to completed-request totals and adds an end-to-end latency observation, despite receiving no tokens or DONE segment for that request.

The existing CPU fake-engine `BUSYALL` scenario in `serve/test_parallel.py` reproduces this with one successful request and one rejected request:

- HTTP response for the rejected request: 503.
- Rejected history row: `finish="length"`, `prompt_tokens=51`, `output_tokens=0`, `engine_generated=null`.
- Completed-request total: 2, although only one request generated a reply.
- `/metrics` in Prometheus format exports this total as `vllm:request_success_total` and includes the unprocessed prompt tokens.

Expected: preserve the 503 and `Retry-After` contract, retain an error record, and exclude a request rejected before admission from completed-request token and latency accounting. Subsequent successful requests should still be admitted and counted normally.

### Strata version, GPU, OS

Upstream main at `477ac13`, Windows, Python 3.12.7. Reproduced using the existing CPU fake batch engine; no GPU or model weights are required.

### Engine log

The fixture repeatedly emits `ERR BGEN: no such free slot (--batch 2) or a bad max_new`. The server exhausts the existing bounded retries and returns 503, as designed by #1994. This report concerns the resulting accounting, not that admission or retry policy.

En el sitio

Enlaces a install, modelos, releases.