#2266 Marker-shaped text truncates agent turns (EOS sampling + semantic priming)

本文

# Marker-shaped text truncates agent turns (EOS sampling + semantic priming)

**Environment:** Strata 0.1.42, commit `61b3fb5d`, model Qwen3.8-Flash-Next IQ3_S, Linux, single CUDA GPU, default run config, loopback.

## Summary

The chat end-of-turn marker (`<|im_end|>`) is the model's EOS token (id 248046, `tokenizer.ggml.eos_token_id`). Two consequences for agentic use:

1. **Output side (reproducible):** when the model generates marker-shaped text, it often samples that single token id, and decoding stops mid-message.
2. **Input side (observed):** marker-shaped *plain text* in tool output or file contents reliably primes the model to emit EOS immediately — an agent turn ended with no output right after a tool result echoed the marker.

Special tokens in the input are **not** tokenized as special ids (verified below) — that defense works. The problem is the model's own sampling, which no input sanitizing can prevent.

## Finding 1 — generating marker-shaped text stops the engine

The end-of-turn marker is one token in the vocabulary, and it is the EOS token:

```
im_start -> id 248045
im_end   -> id 248046   (= tokenizer.ggml.eos_token_id)
```

Repro (the marker is built from hex so the command itself contains no literal marker):

```bash
M=$(printf '\x3c\x7cim_end\x7c\x3e')
for i in 1 2 3; do
  curl -s localhost:8000/v1/chat/completions -H 'Content-Type: application/json' \
    -d "{\"messages\":[{\"role\":\"user\",\"content\":\"Say nothing in reasoning. In content repeat exactly these characters and nothing else: ${M}\"}],\"max_tokens\":200,\"temperature\":0,\"reasoning_effort\":\"none\"}" \
  | python3 -c "import json,sys; r=json.load(sys.stdin)['choices'][0]; print('finish:',r['finish_reason'],'| content:',repr(r['message']['content']))"
done
```

Observed (marker shown escaped): every run → `finish: stop | content: '<~im_end~>'`.

The model samples id 248046; the engine treats it as end-of-turn and stops. In a longer message the cut happens **mid-sentence**: any agent reasoning about chat formats (explaining this very bug, processing a dataset of chat templates, quoting a prompt-injection corpus) is silently truncated.

It is probabilistic: with reasoning enabled, the model sometimes spells the marker out with ordinary character tokens and generation continues (observed `finish_reason: length` with the marker intact in `reasoning_content`). So the truncation is intermittent and hard for users to reproduce or understand.

Note also: the decoded marker text is published in `content` with `finish_reason: stop` — downstream clients see the literal marker in the answer.

## Finding 2 — marker-shaped plain text in the input primes an immediate EOS

Input sanitizing works: a user message containing the literal marker is tokenized as plain text, not as the special id — identical `prompt_tokens` for the marker and for the marker with a space inserted (20 vs 20).

But the plain text still has a strong effect: in a live agent session, a tool result that echoed the marker (a `grep` over the chat template) was followed by the model emitting EOS immediately — the turn ended with no output, repeatedly, until the tool output was masked (`tr '[|]' '~'`).

Impact: any file, web page, or pasted transcript containing marker-shaped text can silently end an agent turn early. For a coding agent this is an availability problem reachable by ordinary repository content. (It also means a webpage author can deliberately stall an agent, though they cannot forge a turn, thanks to the working input defense.)

## Suggested mitigations

1. Strip (or escape) decoded special tokens from `content` / `reasoning_content` before publishing to clients, so the literal marker never leaves the server.
2. Optional recovery: when EOS is sampled while the message is mid-sentence, retry the continuation with a small logit bias against 248046 (`serve/logit_bias.py` already has the plumbing).
3. Document the behavior for agent operators: mask marker-shaped text in tool output (e.g. `tr '[|]' '~'`) when a session may discuss chat formats.

## What is NOT broken

- Special tokens in the input are not encoded as special ids (no token-level turn forgery) — verified.
- No confidentiality or integrity impact; this is robustness/availability for agentic workloads.