Pull requests / #3138
#3138 serve: read llama.cpp's n_predict, including the prompt-only 0 (#2980)
open · @ctr-137 · 0 comments · View on GitHub
Server & APIModels & quantsDocumentationWindows
Description
Issue: #2980 ## Summary `/v1/chat/completions` read only `max_tokens` / `max_completion_tokens`. llama.cpp's web UI sends a follow-up with `n_predict: 0` after every reply to evaluate the prompt into the conversation cache; Strata treated 0 as "the rest of the context" and generated a second full answer (~20k tokens, 4-5 minutes of GPU on the reporter's R9700). Because Strata serves one request at a time, every other client waited behind it, and the next user message re-read the whole conversation because the prompt-only call never completed. llama.cpp's server README defines the field this way: "When 0, no tokens will be generated but the prompt is evaluated into the cache." ## What changed `n_predict` is honoured on `/v1/chat/completions` when the request has neither `max_tokens` nor `max_completion_tokens`: - **`0`** evaluates the prompt through the normal generation path and generates nothing. The answer is empty with `finish_reason: "length"` - the shape llama.cpp returns for this - and `usage.completion_tokens` is 0. Streaming and non-streaming. - **`N > 0`** is the output cap. - **`-1`** (or absent) leaves the rest of the context, as before. An OpenAI `max_tokens` / `max_completion_tokens` in the same request wins over `n_predict`, so an existing client keeps exactly the behaviour it had. A non-integer `n_predict` (`"0"`, `true`, `1.5`) is a 400 before inference. One engine note: `GEN` needs `max_new >= 1`, so the prompt-only call samples and drops one token. The prompt is read through the normal generation path, which is what the client's follow-up wanted. The engine's own totals count that one token; the API response reports 0 completion tokens, matching llama.cpp. `docs/DETAILS.md` has one bullet under the API section. ## How this was checked `python -m unittest serve.test_n_predict` - 6 tests, OK: | request | result | | --- | --- | | `n_predict: 0` | 0 completion tokens, `finish_reason: "length"`, empty content, the prompt reached the engine | | `n_predict: 0`, `stream: true` | role chunk, final chunk with `length` and usage 0, `[DONE]`, no content | | `n_predict: 3` | 3 completion tokens, `length` | | `max_tokens: 2` + `n_predict: 7` | 2 completion tokens (the OpenAI field wins) | | `n_predict: -1` | a full answer, `finish_reason: "stop"` | | `"0"` / `true` / `1.5` / `[0]` | 400, the engine is never reached | The full `serve` suite: **854 tests; the only error is `test_winjob.TestWinJob`** (a Windows `tasklist` call returning `None` here), the same one as on unmodified `main` (848 tests). `test_keepawake` is timing-flaky under the full run and passes alone; neither is related to this change. No new skips. ## What I left alone - `/v1/messages` and `/v1/responses`. The report is llama.cpp's chat client, Anthropic requires `max_tokens`, and the Responses API has `max_output_tokens`. - The engine protocol. A true `GEN 0` that samples nothing would be a C++ change; this stays a server-side fix with one sampled token. - The `n_predict` field in `/props` was already reported; it is unchanged. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
Related on this site
Quick links to related pages.