#6612 [Engine] Inkling-2-Mini: replies start with a thinking span that the server discards (hidden tokens, empty answers at small max_tokens)
open · @Vyeche · 0 Kommentare · Auf GitHub
BenchmarksServer & APIAMD / HIPModels & quantsDocumentationLinux
Beschreibung
### Area
[Engine] (speed, kernels, models, the server and API)
### Desktop app, Terminal or source?
Terminal (the `strata` command)
### What happened
With Inkling-2-Mini the model starts its reply with a thinking span although the prompt asks for effort 0, and the server throws that span away. The client sees no `reasoning_content`, waits for tokens it never gets, and pays for them in `completion_tokens`. When `max_tokens` runs out before the answer starts, the reply is empty with `finish_reason: "length"`.
What I measured (SQ-10, temperature 0, default settings, no `reasoning_effort` in the request unless stated):
| Request | completion_tokens | Thinking span | Returned content | reasoning_content |
|---|---:|---:|---|---|
| "Write a Python function that merges two sorted lists…" (41-token prompt), `max_tokens` 1500 | 767 | about 430 tokens | 800 characters | empty |
| "Write a short story of about 300 words…" (29-token prompt), `max_tokens` 450, `reasoning_effort: "none"` | 450 | about 227 tokens | 867 characters, cut off | empty |
| "Say hello in five words.", `max_tokens` 30, `reasoning_effort: "none"` | 30 | not recorded | empty, `finish_reason: "length"` | empty |
| A 9,783-token document question, `max_tokens` 300, `reasoning_effort: "none"` (engine with PR #6594 applied) | 300 | not recorded | empty, `finish_reason: "length"` | empty |
The token layout of the first two replies, from a recording of the server-to-engine protocol: `<|message_model|><|content_thinking|> … <|end_message|><|message_model|><|content_text|> …`. The rendered prompt was `<|message_system|><|content_text|>Thinking effort level: 0<|end_message|><|message_user|><|content_text|>…<|end_message|>`.
It is the model and not the GPU backend: with the same prompt ids, `strata-inkling --cpu --prompt-ids … --gen 6` and the HIP backend both print `200001,200008,…` first, which are `<|message_model|>` and `<|content_thinking|>`.
Where the text goes: `serve/inkling_dialect.py`, `_emit_typed`, returns nothing in the `think` state ("Inkling-2-Mini has no thinking mode (off / "none" only): silently discard any raw <|content_thinking|> span"), and `normalize_effort` / `template_kwargs` always set effort `none`. The model's own `chat_template.jinja` maps `reasoning_effort` to a scale (`none` 0.0 up to `max` 0.99, default 0.9), so the model does have thinking levels.
What I expected: either the thinking span arrives as `reasoning_content` (the dialect's `feed_channels` already has a `"reasoning"` kind), or it does not count against `max_tokens`, or the docs say that Inkling-2-Mini replies carry hidden tokens. I recorded the token layout on SQ-10 only; SQ-15 and SQ-20 were not checked for this.
One effect on published numbers: decode tok/s for Inkling-2-Mini counts the discarded tokens, so the visible text arrives more slowly than the figure says (about half in the two replies above).
### Smallest steps or command that reproduce it
```sh
strata launch --family inkling-2-mini --model sq-10 --context 32k --yes --no-tui --port 8081
curl -s http://127.0.0.1:8081/v1/chat/completions -H "Content-Type: application/json" -d '{"model":"x","messages":[{"role":"user","content":"Say hello in five words."}],"max_tokens":30,"reasoning_effort":"none","temperature":0}'
```
Result: `"content": ""`, `"finish_reason": "length"`, `"completion_tokens": 30`.
### Reproduction status
Reproduced locally
### Strata version
Strata 0.2.19 · strata-inkling 0.1.65 (hip-rx9070 package; the 9,783-token row with PR #6594 applied, source build)
### Graphics card, memory and system
Radeon RX 9070 XT 16 GB (gfx1201), 92 GB RAM, Ryzen 9 9950X3D, Pop!_OS, Linux 7.2
### Model, size and settings
Inkling-2-Mini SQ-10, 32K context, pictures on (CPU reader), DSpark drafting (default)
### Logs
No error is logged; the server reports the request as done.
This report was prepared with the help of an AI assistant; the measurements were run on the hardware above.
Mehr auf der Site
Links zu Install, Modellen, Releases.