Pull requests / #3332
#3332 [Other] add legacy /v1/completions for raw-prompt clients (#2277)
open · @kossum · 0 comentarios · En GitHub
Setup & installServer & APIAMD / HIPNVIDIA / CUDAModels & quantsWindowsLinux
Descripción
## Problem and result Clients that build their own prompts need `/v1/completions`, but Strata has no legacy completions endpoint. Sending those prompts through the chat API would apply the chat template again. This adds `/v1/completions` for string prompts, with streaming and non-streaming responses in the OpenAI `text_completion` format, including token usage. ## Related discussion #2277, #2278. This replaces #2278 after the tracker reset. The `/reopen` format check passed twice, but GitHub returned "Validation Failed" both times. The bot suggested opening a new PR and linking the old one. The earlier review and fixes are in #2278. ## Approach The handler reuses the existing generation path without rendering a chat template. `max_tokens: 0` or a negative value uses the remaining context. If the caller explicitly disables thinking, the handler appends the template's no-thinking tail. Otherwise it leaves the prompt alone. Streamed text is sent once. With `echo: true`, the prompt goes out once at the start, and the finish chunk has empty text. The tests concatenate the chunks and compare them with the non-streaming reply, with echo on and off. The structured-output error handler stays in the chat endpoint. A failed streamed response sends an SSE error followed by `[DONE]`, rather than a second HTTP response inside the stream. Engine errors after streaming starts are also returned as SSE errors. ## Validation Tested `7e08755f`, rebased onto main `e88ee39d` (0.2.0-beta). **Passed:** On Windows with Python 3.13.2 and jsonschema installed: ```sh uv run --no-project --with regex --with jinja2 --with jsonschema python -m unittest serve.test_server serve.test_structured serve.test_monitor ``` 368 tests, OK, 3 skipped. On Linux in a Python 3.11 Docker container, using a dedicated venv with only the pinned `requirements.txt` packages: ```sh python tools/minimal_server_smoke.py --report /tmp/minimal-server-smoke.json ``` The smoke test reported `"passed": true`. `git diff --check upstream/main` passed too. **Failed:** In the same Linux venv: ```sh python -m unittest discover -s serve -p 'test_*.py' ``` 888 tests, 2 failures, 18 skipped. The failures were: - `test_lifeline.Lifeline.test_the_portable_watcher_ends_the_child_when_the_server_is_killed`: the test still sees the child after killing the server. - `test_server.VisionTempFiles.test_work_dir_has_no_space_when_temp_has_one`: the temporary path contains a space. I reran both tests on clean main `e88ee39d` with the same container and dependencies. Both failed with the same assertions there. This PR doesn't change either path. **Not run:** - Live-model tests on this revision. The live checks mentioned in #2278 were on an older revision. - CUDA, HIP, SYCL or Metal builds; no native backend source is changed. - A byte-for-byte comparison of the existing endpoints against the release. ## Limits The endpoint accepts one string prompt and supports only `n=1` and `best_of=1`. Suffix and logprobs are rejected. `stream_options` accepts only `include_usage`, with streaming enabled. Usage is still on the finish chunk rather than a separate `choices: []` chunk, so the OpenAI streaming-usage format isn't fully implemented.
En el sitio
Enlaces a install, modelos, releases.