Pull requests / #3356
#3356 [Other] Add assistant prefill support for chat completions
open · @j-luwierski · 0 comments · View on GitHub
Setup & installServer & APIAMD / HIPNVIDIA / CUDAModels & quantsLinux
Description
Problem and result: Strata's chat APIs could not continue generation from a partially written assistant turn. Requests can now provide an optional assistant_prefix; Strata prefills that text and returns only the generated continuation. When the field is absent, null, or empty, existing behavior is unchanged. Related discussion: Replacement for [#1189](https://github.com/Niko1221/Strata/pull/1189). Its /reopen check passed, but GitHub would not reopen that closed pull request. Approach: Render the supplied prefix as the unfinished final assistant turn in the Python serving layer. Preserve the prefix text and existing tool, structured-output, streaming, reasoning, and context handling. Document the request field and add regression coverage. Validation (revision baccd430, rebased on main e88ee39d / 0.2.0-beta; Linux): Passed: - git diff --check origin/main...HEAD — clean. - python3 -m py_compile serve/frontend.py serve/server.py — passed. - uv run --with-requirements requirements.txt python -m unittest serve.test_server.AssistantPrefix — 20 tests, OK (Python 3.12.14). - uv run --with-requirements requirements.txt python -m unittest serve.test_server — 361 tests in 104.6 seconds, OK (Python 3.12.14). Failed: - python3 -m unittest serve.test_server.AssistantPrefix with the system Python — could not import jinja2 because it is not installed in that interpreter. The same tests passed using the repository's pinned dependencies via uv. Not run: - Live model/GPU validation on the rebased revision. This change does not touch C++, CUDA, HIP, or SYCL code. Limits: The rebased revision has mock-engine test coverage; hardware behavior has not been revalidated on this revision.
Related on this site
Quick links to related pages.