改动 / #3138

#3138 serve: read llama.cpp's n_predict, including the prompt-only 0 (#2980)

open · @ctr-137 · 0 评论 · 去 GitHub 看

Server & APIModels & quantsDocumentationWindows

说明

Issue: #2980

## Summary

`/v1/chat/completions` read only `max_tokens` / `max_completion_tokens`. llama.cpp's web UI sends a follow-up with
`n_predict: 0` after every reply to evaluate the prompt into the conversation cache; Strata treated 0 as "the rest of
the context" and generated a second full answer (~20k tokens, 4-5 minutes of GPU on the reporter's R9700). Because
Strata serves one request at a time, every other client waited behind it, and the next user message re-read the whole
conversation because the prompt-only call never completed. llama.cpp's server README defines the field this way:
"When 0, no tokens will be generated but the prompt is evaluated into the cache."

## What changed

`n_predict` is honoured on `/v1/chat/completions` when the request has neither `max_tokens` nor
`max_completion_tokens`:

- **`0`** evaluates the prompt through the normal generation path and generates nothing. The answer is empty with
  `finish_reason: "length"` - the shape llama.cpp returns for this - and `usage.completion_tokens` is 0. Streaming
  and non-streaming.
- **`N > 0`** is the output cap.
- **`-1`** (or absent) leaves the rest of the context, as before.

An OpenAI `max_tokens` / `max_completion_tokens` in the same request wins over `n_predict`, so an existing client
keeps exactly the behaviour it had. A non-integer `n_predict` (`"0"`, `true`, `1.5`) is a 400 before inference.

One engine note: `GEN` needs `max_new >= 1`, so the prompt-only call samples and drops one token. The prompt is read
through the normal generation path, which is what the client's follow-up wanted. The engine's own totals count that
one token; the API response reports 0 completion tokens, matching llama.cpp.

`docs/DETAILS.md` has one bullet under the API section.

## How this was checked

`python -m unittest serve.test_n_predict` - 6 tests, OK:

| request | result |
| --- | --- |
| `n_predict: 0` | 0 completion tokens, `finish_reason: "length"`, empty content, the prompt reached the engine |
| `n_predict: 0`, `stream: true` | role chunk, final chunk with `length` and usage 0, `[DONE]`, no content |
| `n_predict: 3` | 3 completion tokens, `length` |
| `max_tokens: 2` + `n_predict: 7` | 2 completion tokens (the OpenAI field wins) |
| `n_predict: -1` | a full answer, `finish_reason: "stop"` |
| `"0"` / `true` / `1.5` / `[0]` | 400, the engine is never reached |

The full `serve` suite: **854 tests; the only error is `test_winjob.TestWinJob`** (a Windows `tasklist` call returning
`None` here), the same one as on unmodified `main` (848 tests). `test_keepawake` is timing-flaky under the full run
and passes alone; neither is related to this change. No new skips.

## What I left alone

- `/v1/messages` and `/v1/responses`. The report is llama.cpp's chat client, Anthropic requires `max_tokens`, and the
  Responses API has `max_output_tokens`.
- The engine protocol. A true `GEN 0` that samples nothing would be a C++ change; this stays a server-side fix with
  one sampled token.
- The `n_predict` field in `/props` was already reported; it is unchanged.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

本站相关内容

相关页面的快捷入口。