Pull requests / #3291
#3291 [Engine] serve: report timings on /v1/responses
open · @pschaub · 0 commentaires · Sur GitHub
Server & APIAMD / HIPNVIDIA / CUDAWindowsLinux
Description
## Problem and result /v1/chat/completions returns top-level `timings` (#107), but /v1/responses (#451) does not, although server.py already computes them for every request. Now the final `response.completed` / `response.incomplete` event and the non-streaming reply include the same `timings` object. ## Related discussion Follow-up to #107 (based on #83) for /v1/responses. ## Approach `_with_timings` in serve/responses.py copies `done["timings"]` onto the final event; `collect()` keeps it on the non-streaming reply. Nothing else changes. ## Validation **Passed:** - `python -m unittest discover -s serve -t . -p "test_*.py"` on this branch (one commit on main e88ee39), Windows 11, Python 3.12: 882 tests OK, 20 skipped. The same command on main e88ee39 without the change: 881 tests OK, 20 skipped. - The new `UsageAndStatus.test_responses` fails without the change. - Live check on main e88ee39 (Windows 11, CUDA): the final event of streamed /v1/responses requests carries timings; without the change it does not. **Failed:** none observed **Not run:** - Linux and macOS. - The non-streaming /v1/responses reply was checked by the unit test only; the live check used streaming. - No CUDA, HIP, SYCL or Metal build: the change is Python only. ## Limits `response.failed` carries no timings. --- Created with Claude Opus 5.5. The result was verified by a human (me) in llama-swap: prompt and decode speeds now show for /v1/responses requests.
Sur le site
Liens install, modèles, releases.