Pull requests / #454

#454 serve: honor stop and stop_sequences

closed · @zuraiz-anjum · 0 comments · View on GitHub

Server & APIDocumentation

Description

The server ignored OpenAI `stop` and Anthropic `stop_sequences` without an error, so a client got the text past its stop string and stop_reason "end_turn". With the mock engine scripted to say "alpha END beta gamma":

Before:
OpenAI stop=['END'] -> 'alpha END beta gamma' stop
Anthropic stop_sequences=['END'] -> 'alpha END beta gamma' end_turn None

After:
OpenAI stop=['END'] -> 'alpha ' stop
Anthropic stop_sequences=['END'] -> 'alpha ' stop_sequence END

Service.run now cuts the answer text at the first stop string and leaves it out. Only the end of the text that could still be the start of a stop string is held back, so a stop split across tokens or streamed chunks is caught, and anything that turns out not to be a stop goes out as normal. On a match run() breaks out of the generation loop the same way a stop token does, so gen.close() sends STOP and the engine does not keep decoding. No engine change was needed. OpenAI takes a string or a list of up to 4 and Anthropic a list; a bad value is a 400. Anthropic sets stop_reason "stop_sequence" and stop_sequence to the matched string, streamed and not. Only answer text is matched, not thinking or tool call arguments. docs/DETAILS.md lists stop strings with the other honored fields.

New tests in serve/test_server.py (StopStrings) cover both APIs, streamed and not, several stops, a stop split across tokens, a near miss prefix, no stop, an empty list, bad values, and that the engine stopped early.

python -m unittest serve.test_server serve.test_mcp serve.test_lifecycle serve.test_monitor serve.test_detok serve.test_structured serve.test_winjob plus all tools.test_setup_* modules: 270 tests OK (7 skipped).

One edge I left alone: if a reply hits a stop string and also has a finished tool call, OpenAI still reports finish_reason "tool_calls". Easy to change if you'd rather stop win.

Related on strata.com

Editorial links to help you install, pick models, or read release notes — not part of the upstream thread.