#6437 [Engine] Replies end mid-word with finish_reason "stop" at ~140K context (int8 KV, MTP on)
open · @fcoberrios · 0 Kommentare · Auf GitHub
BenchmarksServer & APINVIDIA / CUDAModels & quantsWindows
Beschreibung
**Reproduction status:** Reproduced locally
**Strata version:** Strata 0.2.16 · engine 0.1.62 (prebuilt, CUDA 13.0, commit 1d295787b365)
**How it runs:** Source (`serve/server.py --engine strata --config ...`)
**Hardware:** RTX 6000 Ada 48 GB (driver 610.88) · 128 GB RAM · Ryzen 9 5950X · Windows 11
**Model:** Qwen3.8-Flash-Next GSQ-RCO IQ3_S (2-part GGUF, PLE in the second file) · 262K context · pictures on
### What happens
With a ~140K-token prompt, about 1 in 9 replies stops in the middle of a word, after reasoning that ended normally. The server reports `finish_reason: "stop"` and logs `done: N tokens ... (stop, cancel=False)`. There is no error or warning, and no limit was reached: `max_tokens` was 24000 and the reasoning budget was 16384.
The ends of the three cut replies (the answers are in Spanish), with the number of tokens generated:
```text
12,414 tokens: ...nothing repeated."* **Condic
17,458 tokens: ..._transitions`) son diferentes, y la secu
16,728 tokens: ...| 3 | SLA y DST | **Alta** — <- inside a Markdown table
```
The cut does not happen at a fixed length: other replies to the same prompt run to 23,808 tokens and end normally. With short prompts (1-3K tokens, 270 coding replies with the same sampling) it never happened.
### Numbers
| KV | Context | Replies | Cut mid-word |
|---|---|---:|---:|
| int8 | 139.6K prompt | 27 | **3** |
| fp16 | 139.6K prompt | 3 | 0 |
| k8v4 | 139.6K prompt | 3 | 0 |
| int8 | 1-3K prompt | 270 | 0 |
The fp16 and k8v4 samples are too small to rule them out. The cuts happened at temperatures 0.3, 0.6 and 0.8 (top_p 0.95, top_k 20), so temperature does not look like the cause.
### Settings
- Engine args: `--spec 4 --spec-min-p 0.5 --mtp <rt> --max-context 262144 --kv int8 --vision --vram-reserve-mib 2560 --conversation-cache-mib 32768 --conversation-cache-slots 8 --expert-cache auto --prefill auto`
- Config: `"draft_vocab": "latin"`, `"sampling": {"presence_penalty": 0}`, `"reasoning_close_retry": true`, `"env_to_user": true`
- Request: `/v1/chat/completions`, not streamed, `reasoning_effort: "medium"`, `reasoning_budget_tokens: 16384`, `max_tokens: 24000`
### Log around one cut
```text
[strata] thinking: 14890 of budget 16384 (max 24000) tokens, 168.4 tok/s, 129 s
[strata] answering: 15050 of max 24000 tokens, 168.3 tok/s, 130 s
...
[strata] answering: 16707 of max 24000 tokens, 167.9 tok/s, 140 s
[strata] done: 16728 tokens in 140 s (167.8 tok/s) (stop, cancel=False), expert cache 99.5% hit
```
### How to reproduce
1. Send 6 messages: a system prompt, a user message of ~125K tokens of filler source files, an assistant ack, a user message with a ~2,400-line Python app, another assistant ack, and the instruction "audit this app for logic bugs".
2. Repeat the same request about 10 times, with a different nonce at the start of the system message so the conversation cache is not reused.
3. Check whether the answer ends in the middle of a word or table row.
I can share the prompt file (about 140K tokens, all synthetic code) if that helps.
### What I have not tested yet
- MTP / speculative decoding off, to check whether a drafted end token gets accepted.
- v0.2.17. I read the commit titles from 0.2.16 to 0.2.17 and found nothing about this.
- More fp16 runs.
The earlier mid-word cuts (#4576, #4137) were caused by end-marker pieces in tokenizers that split `<|im_end|>`. This one looks different: for this model `<|im_end|>` is a single token, and the cuts only appear with a very long context.Mehr auf der Site
Links zu Install, Modellen, Releases.