#2519 gfx1201 (R9700, HIP): the `!` degeneration is #879 — GDN state overflow (max |x| 2.26e+32) at 16K–93K; 0.1.44.1's guard catches and recovers it on this box

本文

Following the maintainer's note in #1468 ("if this still happens on 0.1.44, leave a comment here … a new issue that links this one works too"). This is a single-card gfx1201/HIP data point, and it comes with a mechanism trace plus a confirmation that 0.1.44's NaN guard catches this fault **on a machine where your own 1,870-request R9700 run could not fire it**.

## Environment

| | |
|---|---|
| GPU | 1x Radeon AI PRO R9700 (gfx1201, RDNA4), 32 GB |
| Host | 62 GiB RAM, Linux 6.17, NVMe; no swap in use during any run |
| Model | Qwen3.8-Flash-Next **GSQ-RCO IQ3_XXS** (`ISTA-DASLab/…`, rev `ed59f920`; shard 1 47,039,860,096 B, sha256 `219ea929900dfa9ef091f3aa473fdba6874b65fcb36526d7d851ac9e95856d15`) |
| Build | local HIP build against the **ROCm 7.10 "TheRock" userspace** (setup's venv wheels, not system ROCm) |
| Versions | most measurements on **0.1.42**; the guard confirmation below is on **0.1.44.1** (same box, same build chain, upgraded today) |
| Args | `--max-context 131072 --kv int8 --prefill auto:32768 --spec 4 --spec-min-p 0.5 --mtp rt --vram-reserve-mib 1000` |
| Env | `STRATA_PLE_BATCH=0`; WMMA off; `STRATA_PF_FUSED` unset (0.1.42 default) / explicitly `=0` on 0.1.44.1 |

## 1. The `!` degeneration is the non-finite family — traced here on 0.1.42

`STRATA_DBG_NAN=1` already exists in 0.1.42, so no guard is needed to see it. Same config, two 16K requests, one degenerate and one clean:

```
# degenerate (reply = 40 x '!')
strata dbg: layer 0 (mmq 1, types 17/20, 0 experts): non-finite GU 0 H -1 Dm 0 bo 33226240 of T 17075
strata dbg: prompt end: last residual row 10240 non-finite (max |x| 0); PLE history 0 non-finite (max |x| 21.1);
            GDN state 0 9264 non-finite (max |x| 2.26e+32)
strata dbg: verify window at position 17075, row 0: 248320 of 248320 logits non-finite (token out 0)

# clean request, same config
strata dbg: prompt end: last residual row 0 non-finite (max |x| 5.13); PLE history 0 non-finite (max |x| 12.4);
            GDN state 0 0 non-finite (max |x| 3.34)
```

- **`248320 of 248320 logits non-finite`** — the same count the reporter in #1208 got on gfx1030 (whole vocab).
- **`GDN state … 9264 non-finite (max |x| 2.26e+32)`** — the recurrent state's magnitude has run away to 1e32 before the residual goes non-finite; the clean request prints **0** for both fields. So here the chain is: *GDN state overflow → residual row non-finite → all logits NaN → sampler collapses to token 0 (`!`)*.
- This fires at a **16K** prompt, while the same box reads 63K/76K/85K cleanly — so on this hardware it is not a context-capacity effect.
- Cost of the instrument, for anyone reusing it: `STRATA_DBG_NAN=1` takes a 16K-prompt prefill from ~1,400 tok/s to ~310 tok/s (~4.5x), and it prints informational (all-zero) lines on passing requests too.

## 2. On 0.1.44.1 your guard catches exactly this, and the retry recovers it

Interleaved arm, 40K cold needle prompts, fresh nonce per request, engine restarted per arm, with our amplifier `STRATA_PF_FUSED=1` (see §5):

```
non-finite logits (#879) in a prompt window at position 38621 (row 0): its tokens are not emitted,
the request ends (DONE nonfinite) and the next request reads its prompt from token 0
non-finite logits (#879): dropped the live session, 2 checkpoints, 0 parked conversations and 0 idle slots …
```

The three probes in that arm **all returned the correct answer** (via the retry), and the log line names the position. On 0.1.42 the same fault was emitted as 40 `!` characters with no log line at all, so from a user's point of view the guard turns silent corruption into a visible, recoverable event. Thank you for #1892.

## 3. Server-side behaviour worth documenting

After a degeneration, re-sending the **same** prompt is served from the prefix cache (`prompt 38438 tokens = 38431 reused + 7 read in 38 ms`) and **degenerates again**; a **different** prompt sent immediately after, same engine, answers correctly. So the poisoned state follows the conversation/prefix rather than the engine — consistent with #2452's retry-from-0 and with #879's "session-scoped, not engine-global". Users may be wasting a full re-read by retrying the same request; a new conversation recovers without a reload.

## 4. Two levers that do nothing here (so other 32 GB / gfx12 users don't spend the time)

- **`STRATA_HIP_PROMPT_F16=1`** (#1208's fix on gfx1030): interleaved 8 rounds x 2 arms, 40K, `STRATA_PF_FUSED=1` in both arms, on 0.1.42 — **3/8 vs 2/8, Fisher one-tailed p = 0.50**. It does not transfer to single-card gfx1201.
- **`STRATA_KV_STAGE_OWN=1`**: `take_stage()` returns unless the session is KV-streaming, so on a **KV-resident** config the variable has no object at all (our logs never contain `KV streaming:`). We then built the condition it needs (`--kv-resident 32768`) and first verified the lever acts — `the prompt path borrows 8066 → 7988 CUDA0 cache slots (13.14 → 13.01 GiB)`, the 13.01 GiB being exactly the resident-run figure your comment in `prefill.cpp` predicts — and the A/B is **null: 1/8 vs 1/8 (Fisher p = 0.77)**, 0 engine faults in the 21 requests of that configuration. Worth a line in the docs: on a 32 GB card with resident KV this arm can only produce a false negative.

## 5. The amplifier, for anyone trying to reproduce

`STRATA_PF_FUSED=1` (0.1.42, opt-in there): at 40K it raises the degeneration rate from 0/8 (baseline) to **3/8 (37.5%)** in an interleaved arm — consistent with our earlier 6/18 vs 0/8. That is why we pin `STRATA_PF_FUSED=0` in production and why the fact that 0.1.43 turned it on by default for IQ3_XXS matters to us.

## 6. And the other signature, for the record

`prefill copy_i32: an illegal memory access` at ~93K tokens (91,511, frozen at `65,536 of 91,511`), ≈1/13 requests on 0.1.42 with the production config; 63K/76K/85K read cleanly. We have only one 93K pass on 0.1.44.1 so far, so we cannot say anything about whether that changed.

## 7. What we could not test

Our engine is built against the **TheRock 7.10 wheels**. The follow-up comment in #1208 reports the default batched MMQ path as finite on ROCm 10 system packages and NaN with the TheRock 7.10 wheels, which points at the ROCm side rather than the engine. We have not A/B'd ROCm userspaces here (this box also serves two other engines that depend on the current setup). If that comparison is worth having, say which userspace and we will build it.

We can also bisect, or dump token ids around a failing position, or run any specific env/arm on gfx1201 — the config above is the exact one. If `STRATA_DBG_NAN`-style counters at the GDN update are easier than at the logits, that would localise the overflow source faster than we can from outside.