Pull requests / #3291

#3291 [Engine] serve: report timings on /v1/responses

open · @pschaub · 0 comentarios · En GitHub

Server & APIAMD / HIPNVIDIA / CUDAWindowsLinux

Descripción

## Problem and result
/v1/chat/completions returns top-level `timings` (#107), but /v1/responses (#451) does not, although server.py already computes them for every request. Now the final `response.completed` / `response.incomplete` event and the non-streaming reply include the same `timings` object.

## Related discussion
Follow-up to #107 (based on #83) for /v1/responses.

## Approach
`_with_timings` in serve/responses.py copies `done["timings"]` onto the final event; `collect()` keeps it on the non-streaming reply. Nothing else changes.

## Validation
**Passed:**
- `python -m unittest discover -s serve -t . -p "test_*.py"` on this branch (one commit on main e88ee39), Windows 11, Python 3.12: 882 tests OK, 20 skipped. The same command on main e88ee39 without the change: 881 tests OK, 20 skipped.
- The new `UsageAndStatus.test_responses` fails without the change.
- Live check on main e88ee39 (Windows 11, CUDA): the final event of streamed /v1/responses requests carries timings; without the change it does not.

**Failed:**
none observed

**Not run:**
- Linux and macOS.
- The non-streaming /v1/responses reply was checked by the unit test only; the live check used streaming.
- No CUDA, HIP, SYCL or Metal build: the change is Python only.

## Limits
`response.failed` carries no timings.

---
Created with Claude Opus 5.5. The result was verified by a human (me) in llama-swap: prompt and decode speeds now show for /v1/responses requests.

En el sitio

Enlaces a install, modelos, releases.