#6620 [Engine] Inkling 2 Mini: empty reply with finish_reason "length" when tools are present
open · @jbatdk · 2 comments · View on GitHub
Setup & installServer & APIAMD / HIPModels & quantsLinux
Description
### Summary
With `tools` in the request, Inkling 2 Mini intermittently returns **no content at all**: no text, no `reasoning_content`, no `tool_calls`. The reply ends with `finish_reason: "length"`, far below `max_tokens`. The tokens are generated (usage shows them) but none reach the client. The identical request without `tools` never fails.
### Environment
- Strata `0.2.19-rocm-gfx120x` (Docker), served via `serve/server.py --engine strata`
- Model: Inkling 2 Mini **SQ-15**, `--max-context 32768 --vram-experts-gb 12.28 --ram-experts-gb 0 --chunk 32`
(`--chunk 32` is the workaround for the gfx1201 prefill NaN issue; no NaN/GPU errors in the log)
- GPU: RX 9070 XT (gfx1201, 16 GB), Linux, ROCm 10.1 on the host
- `drafting: dspark`, `parallel: 1`
### Reproduce
System prompt (~1.2k tokens, plain Danish instructions) + 2 function tools (`search_web`, `fetch_article`, simple JSON-schema params) + this user message:
````
Gennemgå denne docker-compose-fil til en server på mit hjemmenet. Hvilke problemer ser du, og hvordan retter jeg dem?
```yaml
services:
db:
image: postgres:latest
environment:
POSTGRES_PASSWORD: hemmeligt123
ports:
- "5432:5432"
app:
image: minapp:latest
ports:
- "8080:8080"
depends_on:
- db
```
````
`max_tokens: 12000`, `tool_choice: "auto"`, stream or non-stream.
### Results
| Setup | Runs | Result |
|---|---|---|
| system prompt + tools | 9 | **7 failed**: empty reply, `finish_reason: "length"` |
| same, **without tools** | 6+3 | all OK: `stop`, ~900–1300 tokens, normal answer |
Failures come in two forms:
- early stop: `completion_tokens` 344 / 377 / 552, `finish_reason: "length"`, zero content
- runaway: `completion_tokens` = 12000 (the full budget), zero content
Streamed failing example: only the role chunk, then straight to the end:
````
data: {"choices":[{"delta":{"role":"assistant","content":""},"finish_reason":null}]}
data: {"choices":[{"delta":{},"finish_reason":"length"}],"usage":{"prompt_tokens":1405,"completion_tokens":344,...},"timings":{...,"predicted_per_second":69.2}}
data: [DONE]
````
Other prompts in the same tool-enabled session (Dockerfile review, nftables review, web-search questions with real tool calls) work fine. The failing answer naturally contains YAML code blocks, so my guess is that something in the output is taken as the start of Inkling tool markup (`<tool_calls:opensource>` / `<tool_call:opensource>`), and the dialect parser holds everything back waiting for a close that never comes. The early `length` then hides the cause.
### Expected
Either the text is released as content when no valid call follows, or a real call is returned. In no case should the reply be empty with `length` below `max_tokens`.
I can rerun with `STRATA_DEBUG=1` and attach the raw tokens if that helps.Related on this site
Quick links to related pages.