#1827 Chinese output collapses into a token loop after a few hundred tokens

说明

# IQ3_S (unpruned): Chinese output collapses into a token loop after a few hundred tokens; English unaffected — deterministic, 5/5 on greedy

## Summary

Long Chinese answers degrade into a repeated-token collapse partway through generation. The same
model and engine produce clean English at nearly 2x the length. It is fully deterministic on greedy
(5/5 byte-identical outputs), it is not a reasoning loop, and sampling penalties do not help.

The collapse is always in the same semantic cluster — the model gets stuck emitting `颊` and its
English glosses (`cheek`, `blush`, `face`, `扑`) interleaved, after producing several hundred
otherwise-correct tokens.

This is the symptom of #438, but on an **unpruned** IQ3_S pack, so the pruning explanation from that
issue does not apply here.

## Environment

| | |
| --- | --- |
| Model | `ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF`, **IQ3_S**, the original unpruned files |
| Strata | 0.1.41 (`setup.sh` @ `d311345`), engine `BUILD.json`: `{"source":"local","version":"0.1.41","archs":[89],"src":"62eb4907c92a8cb7"}` |
| Engine build | compiled from source, nvcc **13.3.73**, GNU 13.3.0, `CMAKE_CUDA_ARCHITECTURES=89` (no Linux prebuilt exists for any release, so this is the normal path) |
| GPUs | 2× RTX 4090 **48 GB** (modded), driver 595.99.02, layer split `auto` |
| GPU links | CUDA0 **PCIe 4.0 x4** — engine probe `6.1 GB/s -> miss cost x3.27`; CUDA1 PCIe 4.0 x16 — `25.7 GB/s -> x1.00` |
| CPU / RAM | Ryzen 7 3800XT (AVX2, no AVX-512), **61 GB** RAM |
| OS | Ubuntu 26.04 (glibc 2.43), Python 3.14.4 |
| Storage | NVMe SSD for both the pack and the GGUFs |
| Server args | `--pack packs/iq3_s --native …-00001-of-00002.gguf --ple-gguf …-00002-of-00002.gguf --expert-profile data/expert-profile.bin --expert-cache auto --prefill auto --spec 4 --spec-min-p 0.5 --mtp mtp/rt --max-context 131072 --kv int8 --resident-experts` |
| Config keys added | `"reasoning_loop_recovery": "recover"`, `"reasoning_budget_tokens": 8192` (see "Guards" below) |

Experts are **fully resident**: the engine log reports
`resident RAM: 0.00 GiB of experts in RAM, 0 exchanged with the VRAM tier, 0 blob reads from the file`
and start-up filled `12288 experts, 22.14 GiB of VRAM` per card (48 MoE layers × 512 experts = 24,576 total,
12,288 per card). So there are no expert file reads and no VRAM/RAM swaps during these runs.

## Symptom

Prompt `你是谁，详细介绍一下`, default sampling (greedy), `max_tokens: 4096`. `finish_reason` is
`stop` — the model ends the turn itself, it is not truncated.

The first ~400 tokens are correct and well formed. The tail of the same answer:

```
… 如果你在学习某个概念，我可以把它讲 face-to-face 颊 cheek颊颊扑 cheek颊颊 cheek face-to-face face-to-face
```

Reasoning for the same request is short (124 characters) and clean.

## Reproduction

```bash
curl -s http://127.0.0.1:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"strata","messages":[{"role":"user","content":"你是谁，详细介绍一下"}],"max_tokens":4096}'
```

Run 5 times back to back: **5/5 byte-identical**, 474 completion tokens, `finish_reason=stop`, same
collapsed tail.

## Measured matrix

`重复度` below is `1 - unique/total` over character 8-grams of the answer (0 = no repetition).

| Case | tokens | `finish_reason` | result | tail |
| --- | --- | --- | --- | --- |
| Chinese, defaults | 474 | `stop` | **collapsed** | `…讲 face-to-face 颊 cheek颊颊扑 cheek颊颊 cheek face-to-face face-to-face` |
| Chinese, `max_tokens: 200` | 200 | `length` | **clean** | `…- 日常问答 - 知识解释 - 写作与润色 - 翻译` (truncated before the collapse point) |
| Chinese, `temperature: 0.8, seed: 7` | 1084 | `stop` | **collapsed** | `…整颊颊颊颊颊横扑颊 face cheeky  face cheecky cheeks cheeks cheeks.` |
| Chinese, `reasoning_effort: "none"` | 685 | `stop` | **collapsed, worse** | `…颊颊颊颊颊颊颊颊颊颊颊颊颊颊颊颊颊颊颊颊颊颊颊颊…` (60+ consecutive `颊`) |
| **English** control — `Who are you? Please introduce yourself in detail.` | **1885** | `stop` | **clean** | `… can be a thoughtful, flexible, and capable assistant when used carefully and effectively.` |
| Chinese, other topic — `介绍一下杭州的历史和文化` | 1436 | `length` | **collapsed** | `… cheek cheek cheek cheek cheek cheek cheek …` |

Two things follow from this table:

* **It is language-specific.** English is clean at 1,885 tokens; Chinese collapses at 429–1,436 tokens
  in every case tried. A general numerical fault (NaN, a miscompiled kernel) would hit English too.
* **It is not a reasoning loop.** Turning reasoning *off* makes it worse (60+ consecutive `颊`), and the
  collapse is in the answer body.

## Sampling penalties do not help

Same prompt, same session:

| Sampling | tokens | result |
| --- | --- | --- |
| baseline | 474 | collapsed |
| `repetition_penalty: 1.05` | 801 | collapsed — `face face face … face颊 cheek` |
| `repetition_penalty: 1.10` | 429 | collapsed — `把复杂概念讲 face颊红脸上的 cheek blush 扑 cheek blush cheek blush` |
| `repetition_penalty: 1.1` + `frequency_penalty: 0.3` | 319 | collapsed — `颊 cheek颊 cheek颊 cheek颊 … cheeks cheeks cheeks` |

The penalties change the *shape* of the collapse, not its occurrence.

## Existing guards do not fire, and why

* `repeat_stop_tokens` (default 256) — the run of `颊` is only ~60 long and is interleaved with Latin
  tokens (`cheek`, `face`), so the single-token run counter never reaches 256.
* `reasoning_loop_recovery: "recover"` and `reasoning_budget_tokens: 8192` — both were set in
  `strata-iq3_s.json` for these tests and neither applies: the guard watches the **reasoning** span,
  while the collapse is in the answer, and the reasoning here is only 124 characters.

So at present nothing in the server bounds this failure mode: `finish_reason` is `stop`, so a client
cannot even tell the answer is corrupt.

## What I ruled out

* **#438 (CJK wrong at IQ1_M, Coder)** — the official explanation was expert pruning, with the fix
  being "use the unpruned sizes (Q2_0, IQ2_XS, IQ3_S)". This pack *is* the unpruned original
  (`IQ3_S/`, both shards, sizes verified against the repo), and the symptom reproduces anyway. The
  collapse signature (`我是通（通通通通通…`) is also the same family as ours.
* **#137 (draft_vocab has almost no CJK)** — checked the shipped file: `data/draft_vocab.bin` is
  425,196 bytes, which matches the 106,299-id post-fix size, not the 40,525-id broken one. Also MTP
  acceptance on Chinese in these runs is 0.76–0.86, not the 0.22 of #137.
* **#879 (all-logits-NaN) / #892 (CUDA miscompile)** — both are described as making *every* answer
  wrong. Here English is clean at 1,885 tokens and the Chinese answers are correct for the first few
  hundred tokens, so the logits are not globally broken.

## Variables I am disclosing about this setup

I do not know whether any of these matter, but they are unusual and may help triage:

1. **Two-GPU layer split with `--resident-experts`**, experts 100% resident, `0 blob reads from the
   file`, `0 exchanged with the VRAM tier`. I have no data point for this model on a single GPU yet.
2. **Asymmetric links** — CUDA0 is physically x4 (6.1 GB/s, engine's own `miss cost x3.27`) against
   CUDA1 at x16. The split search does see it (`CUDA0 link 6.1 GB/s -> miss cost x3.27`).
3. **The engine binary was compiled on a different machine** and copied over (that build machine was
   a Haswell Xeon, also AVX2-only, same CUDA 13.3.73, same `archs=[89]`). I can recompile on the
   target if that is worth excluding.
4. `--max-context 131072 --kv int8`, MTP `--spec 4 --spec-min-p 0.5` with the fetched MTP adapter.

## Questions

1. Is the answer-body collapse (as opposed to the reasoning loop of #728 / #1622) a known failure
   mode? I could not find an open issue that matches "correct for N hundred tokens, then a
   repeated-token collapse, CJK only, greedy, unpruned".
2. Is there a knob to bound it? `repeat_stop_tokens` counts a *single* token's run, so an interleaved
   `颊 / cheek / face` cycle is invisible to it — something like the `repeat_stop_period` discussed in
   #1613 would catch this shape.
3. Would you like me to run a control on the same engine with a different size (IQ2_XS or
   UD-Q4_K_XL) to separate "IQ3_S dequant path" from "engine-wide CJK path"? I can also recompile on
   the target host, or bisect CUDA versions if #892's miscompile family is still open on sm_89.