Pull requests / #3356

#3356 [Other] Add assistant prefill support for chat completions

open · @j-luwierski · 0 comentarios · En GitHub

Setup & installServer & APIAMD / HIPNVIDIA / CUDAModels & quantsLinux

Descripción

Problem and result: Strata's chat APIs could not continue generation from a partially written assistant turn. Requests can now provide an optional assistant_prefix; Strata prefills that text and returns only the generated continuation. When the field is absent, null, or empty, existing behavior is unchanged.

Related discussion: Replacement for [#1189](https://github.com/Niko1221/Strata/pull/1189). Its /reopen check passed, but GitHub would not reopen that closed pull request.

Approach: Render the supplied prefix as the unfinished final assistant turn in the Python serving layer. Preserve the prefix text and existing tool, structured-output, streaming, reasoning, and context handling. Document the request field and add regression coverage.

Validation (revision baccd430, rebased on main e88ee39d / 0.2.0-beta; Linux):

Passed:
- git diff --check origin/main...HEAD — clean.
- python3 -m py_compile serve/frontend.py serve/server.py — passed.
- uv run --with-requirements requirements.txt python -m unittest serve.test_server.AssistantPrefix — 20 tests, OK (Python 3.12.14).
- uv run --with-requirements requirements.txt python -m unittest serve.test_server — 361 tests in 104.6 seconds, OK (Python 3.12.14).

Failed:
- python3 -m unittest serve.test_server.AssistantPrefix with the system Python — could not import jinja2 because it is not installed in that interpreter. The same tests passed using the repository's pinned dependencies via uv.

Not run:
- Live model/GPU validation on the rebased revision. This change does not touch C++, CUDA, HIP, or SYCL code.

Limits: The rebased revision has mock-engine test coverage; hardware behavior has not been revalidated on this revision.

En el sitio

Enlaces a install, modelos, releases.