#3200 [Engine] OpenAI logprobs / top_logprobs / allowed_token_ids for the CUDA/HIP engine

Beschreibung

Problem and result. `/v1/chat/completions` on the CUDA/HIP engine accepted `logprobs` / `top_logprobs` and returned HTTP 200, but `choices[0].logprobs.content` was always `[]`. A client that ranks candidate tokens from the model's own distribution — constrained decoding, candidate scoring, tool-choice re-ranking — silently got nothing: the request succeeds, the content is correct, only the field is missing, so the failure is invisible unless the caller asserts on the field. After this change, `logprobs: true` with `top_logprobs` 0..20 (the legacy completions form `logprobs: N` is also accepted) returns one entry per token with its top alternatives, and `allowed_token_ids` returns a distribution renormalised over exactly the ids the caller named.

Related discussion. This supersedes #2254, which the 0.2.0-beta tracker reset closed. GitHub will not reopen it (`state cannot be changed. The feat/openai-logprobs branch was force-pushed or recreated.`), so this PR carries the same single commit, rebased onto `main`. #1916 proposed the same feature for CUDA/HIP and is closed. This implementation follows the protocol the SYCL port already speaks (835d30b1) and the Metal port (31f26683), so one server-side reader serves all three backends.

Approach.

- Engine half, `src/program/generate.cpp`: `logprobs=K` and `allowed=a,b,c` are two extra keys on the `--serve` GEN line (an engine that predates them skips them as unknown), and one `LP <logprob> [id:logprob ...]` line is written after each `T <id>` in the same flush. The value comes from the verify window's head logits **before** sampling, penalties and temperature, so it is the model's distribution rather than the sampler's. Off by default: a request without `logprobs` produces the previous output byte for byte, and the per-token cost is one integer compare.
- Server half, `serve/server.py`: `sampling_keys` appends the two keys only when asked; `generate` reads each token's `LP` line before yielding the token (a held line is consumed before the next read, so an engine without LP lines loses no lines, and a `DONE` read while looking for one still ends the request); `LogprobEntries` decodes token bytes incrementally so the entries joined spell the reply; `allowed_token_ids` keeps the named ids and renormalises over them; `structured_chunks` re-aligns the entries when `response_format: json_schema` rewrites the content and returns **no** logprobs rather than a set that does not match.
- Compatibility: additive and opt-in. A request that does not ask for logprobs gets the same bytes as before, and no default changes.

Validation.

Rebased onto `upstream/main` `e88ee39d` (which contains `v0.2.0-beta` `b73dbce3`); head `104becfd`. The diff is 3 files, +570/−8, unchanged by the rebase.

Passed:

- Full CUDA build and link, revision `104becfd`: `cmake --build build-lp --target strata` — Release, `CMAKE_CUDA_ARCHITECTURES=89`, CUDA 13.3 (V13.3.73), MSVC 14.41, Windows 11, RTX 4090. `[154/154] Linking CXX executable strata.exe`, exit 0.
- `python -m unittest serve.test_logprobs` — 17 tests, OK, on `104becfd`.
- Hardware run, same revision and engine (`strata.exe` built from it, model Swift-1.5 IQ3_XXS, `--spec 4 --spec-min-p 0.5 --kv int8 --kv-resident 32768`), `max_tokens: 1`, `temperature: 0`, `chat_template_kwargs.enable_thinking: false`:
  - non-streamed, `top_logprobs: 20` → `logprobs.content` length 1, `top_logprobs` width 20; entry token `A` at -0.1028 with `B -3.121`, `C -3.201`, `D -5.1109`.
  - the same request with `tools` and `tool_choice: "auto"` → entries present, width 20, labels `A/D/C/B`.
  - `stream: true` → the delta carries `logprobs.content` (1 entry, token `A`, -0.0801, top width 20).
- An external client (a separate project) that scores 5 candidate labels from this endpoint: 7 scoring requests in a 30k-token session, 0 errors, 0 disabled calls. Run against the same commit before the rebase (`80494788`); the endpoint behaviour it depends on is the same code path exercised above.

Failed: none observed.

Not run:

- HIP backend build. `src/program/generate.cpp` is shared by the CUDA and HIP builds; only the CUDA configuration was built here (no AMD GPU or toolchain on this machine).
- SYCL backend build. It carries its own implementation (`install_logprobs` in `sycl/serve/server_intel.py`) and this change does not touch it; no Intel GPU or toolchain here.
- `--batch` mode: see Limits.

Limits.

- `--batch` does not return logprobs. `generate_batched` never reads `LP` lines; only the solo/serial path (`generate`) does, so a server started with `--batch` answers `logprobs.content: []`. Giving the batched path the same read-before-yield step is the remaining work if that combination is wanted.
- The reported logprob describes the model's distribution before penalties and temperature, not the post-sampling one.
- With `response_format: json_schema` the returned content may not be spelled by the tokens; in that case no logprobs are returned, by design.
- CUDA/HIP only; the SYCL and Metal implementations are untouched.