#2764 [Bug]: 0.1.44.1 / HIP 2×7900 XTX — 186k-token conversation: cold read OK, every re-read dies in the pipelined verify window (exit −6, then #879); 11/11 clean without the tuning flags

Beschreibung

### What happened

A **long-context conversation cannot be served on 2× AMD RX 7900 XTX (HIP)**: the cold read of a ~185.6k-token document succeeds, and then *every* request that carries that context dies.

The run below is the 13-request conversation an agent actually produces at that context — cold read → 5 continuations → a compaction request → the post-compaction turn → an over-context probe → an "is it alive" probe → 2 concurrent re-reads — against the config I run daily (full args below):

| # | step | result |
|---|---|---|
| 0 | calibration (1,560 tok) | 200 |
| 1 | **cold read, 185,601 tok** | **200** — but only on the *second* document: the first one ended in #879 after 261 s (the harness re-sends the same size with different words). ttft 136.0 s, prefill 1,364.7 tok/s, 128 gen @ 62.8 tok/s |
| 2 | continuation 1/5 | **200** — prompt-cache hit, 1.4 s |
| 3 | continuation 2/5 | **503** — 67.7 s, `the engine stopped unexpectedly (exit code -6)` |
| 4 | continuation 3/5 | **503** — 583.6 s, `#879 non-finite` |
| 5 | continuation 4/5 | **503** — 430.3 s, `#879` |
| 6 | continuation 5/5 | **503** — 485.0 s, `#879`, engine restarted |
| 7 | compaction request | **503** — 336.4 s, `#879` |
| 8 | post-compaction turn | **200** — 224.8 s (after a reload) |
| 9 | over-context probe | **400** — correct |
| 10 | alive probe | **200** |
| 11 | concurrent re-read 1/2 | **503** — 253.9 s, `#879` |
| 12 | concurrent re-read 2/2 | **503** — 582.4 s, `the engine stopped unexpectedly (exit code -6)` |

**5 of 13 requests OK, 5 engine restarts.** It reproduces on every config variant I tried: `--prefill auto:16384` (5/13, ttft 99.6 s), `--prefill auto` (above), and `--adapt-every 0` (5/13, 3 restarts) — always the same shape: **the cold read is served, the first re-read that carries the 185.6k context kills the engine, everything after that is #879.** The two client-visible errors:

```
503 {"error":{"type":"server_error","message":"the engine stopped unexpectedly (exit code -6); the next request restarts it"}}
503 {"error":{"type":"server_error","message":"the model's output became non-finite (NaN) twice in a row (#879); the engine has dropped its cached state - send the request again"}}
```

### The stall the engine reports before the kill

The kill is the engine's own 60 s no-progress watchdog, and its stall report names the stage ("issue #29" in the message): the **pipelined verify window**, at the very tail of the 185,756-token re-read.

```
strata serve: no progress for 60 s during a request (verify window (pipelined): the CPU experts of layer 22) - stopping the engine so the server starts it again (issue #29)
strata serve: stall report (engine 0.1.44.1): stage "verify window (pipelined): the CPU experts of layer 22" for 60 s; 0 layers served since the last finished step (0 = stopped, more = slow)
  VRAM free when this request began: 2806 MiB
  expert pool: epoch 4644, batch epoch 4644: 42 of 42 jobs claimed, 42 done; 13 of 13 workers parked, 13 sleeping; mode 0
  expert pool threads: w0=sleeping w1=sleeping w2=sleeping w3=sleeping w4=sleeping w5=sleeping w6=sleeping w7=sleeping w8=sleeping w9=sleeping w10=sleeping w11=sleeping w12=sleeping; host idle for 60819 ms
  verify window: 6 tokens at position 185615, host at layer step 23; the GPU rang 24; flags: served 23, plan (A) 23, copies (B) 23
  threads waiting on the disk (state D): 0 of 43
  memory: 81016 MiB resident, 0 MiB in swap, 4402 MiB RAM available; 0 major page faults so far
  2 s later:
  (the same lines, host idle for 62819 ms)
strata: released the verify window's GPU waits (#267): the GPU finished in 0 ms
strata: released the verify window's GPU waits (#267): the GPU did not finish within 5 s
```

Note what the stall report itself says: the **expert pool is idle** (42/42 jobs done, all 13 workers parked, host idle for 60 s) and **no thread is waiting on disk** (0 of 43 in state D). The verify window sits at position 185,615 of 185,756 — the last window of the prompt — with the GPU on ring 24 while the host has served 23. This is a GPU-synchronisation hang in the *pipelined* verify window, not expert or I/O starvation.

The #879 NaNs then arrive **after** the first kill, on requests that re-read the same context from token 0 after the reload:

```
strata serve: non-finite logits (#879) in a prompt window at position 185591 (row 0): its tokens are not emitted, the request ends (DONE nonfinite) and the next request reads its prompt from token 0
strata serve: prompt 185756 tokens = 0 reused + 185591 of 185756 read in 125969 ms (1473.3 tok/s), 0 generated in 0 ms (0.0 tok/s), drafts accepted 0 of 0, 6 checkpoints (cancelled)
strata serve: after non-finite logits (#879): dropped the live session, 6 checkpoints, 0 parked conversations and 0 idle slots; reading this prompt from token 0
```

So in this workload the NaN looks like it follows the crash/reload rather than being independent of it — the *first* failure in every run is the watchdog kill.

### Control: the same conversation is clean when the optional flags are dropped

Same box, same model, same 186k context, same 13-request shape. The only difference is the set of *optional* engine flags: I removed `--short-read 260 --pcie-frac 0.00 --spec-min-p 0.70 --pool-workers 13 --adapt-every --adapt-async --adapt-swaps --adapt-decay --conversation-cache-mib --pipeline-windows --layer-split` and all `STRATA_*` env vars, keeping only what the engine needs to load a native (IQ) pack and fit the model (`--pack --native --expert-profile --expert-cache --prefill auto --spec 4 --mtp --max-context 200000 --kv int8 --kv-resident 32768 --vision --vram-reserve-mib 3072` — the engine refuses to start without `--spec/--prefill` for this pack, so a literally flag-free run does not exist here).

| | shipping args | engine-required args only |
|---|---|---|
| requests served (200/400) | 5 of 13 | **11 of 11** |
| engine restarts | 5 | **0** |
| first re-read after the cold read | 503, 67.7 s | **200, 2.9 s** |
| cold read | 200, 185,601 tok, ttft 136.0 s, 1,364.7 tok/s | 200, 186,071 tok, ttft 186.2 s, 999.5 tok/s |
| continuations | 503 ×4 (#879) | 200 ×5, 2.9–4.6 s each |
| compaction request | 503, #879 | **200**, 6.3 s |
| post-compaction | 200 (after a reload) | 200, 4.3 s |
| decode | 62.8 tok/s | 43.1 tok/s |

Every request in that run returned 200 except the intentional over-context probe (400). **So it is not "2 AMD cards cannot do long context"** — the same conversation, on the same two cards, completes with the pipeline and the optional knobs out of the picture. (It is slower: 1,000 vs 1,365 tok/s prefill without the pipeline.)

For completeness, a cross-backend control: the identical 13-request conversation on the same model (NVFP4, single NVIDIA GPU, same box) is **13/13, zero restarts** (cold read 185,598 tok, ttft 82.2 s, 2,257.5 tok/s).

### Environment

| | |
|---|---|
| Strata | **0.1.44.1** (git `296b59d2`, tag `v0.1.44.1`) — HIP source build |
| Engine | `serve/server.py` from the repo, engine `strata --serve` |
| GPUs | 2× AMD RX 7900 XTX (Navi 31, gfx1100), 24 GB each, `--layer-split 24` (also tried `auto`, same failure) |
| Model | Qwen3.8-Flash-Next, **UD-Q4_K_XL native pack** (`unsloth-ud-q4_k_xl`), `--spec 4 --mtp`, vision on CPU |
| CPU / RAM | Intel Core Ultra 7 265K / 96 GB DDR5 |
| OS | Ubuntu 26.04.1 LTS, kernel `7.0.0-38-generic` |
| HIP | `7.16.26385-0000000` |

Engine args (shipping config):

```
--pack <data>/packs/unsloth-ud-q4_k_xl
--native <cache>/qwen38-flash-next-gguf/UD-Q4_K_XL/Qwen3.8-Flash-Next-UD-Q4_K_XL-00001-of-00004.gguf
--expert-profile <data>/expert-profile.bin
--expert-cache auto --prefill auto --short-read 260 --spec 4 --mtp <data>/mtp/rt
--max-context 200000 --kv int8 --kv-resident 32768 --vision --vram-reserve-mib 3072
--pcie-frac 0.00 --spec-min-p 0.70 --pool-workers 13
--adapt-every 1 --adapt-swaps 80 --adapt-decay 0.97 --conversation-cache-mib 8192
--pipeline-windows 2 --layer-split 24
```

env: `STRATA_KV_PREFETCH=1`, `STRATA_MTP_STAGE0=1` (so this is not #2637, which is why `STRATA_MTP_STAGE0` is set), `STRATA_STAGE_TRIM=1`, `STRATA_HIPBLASLT_TUNING=<…>gfx1100-hipblaslt-100401.txt`.

### How to reproduce

Two requests. Any prose document of ~185.6k tokens (my filler is ~1.09 tokens/word) reproduces it; the second request is the one that dies, and a single first request never fails.

```python
import requests

doc = open("185k-document.txt").read()          # ~185.6k tokens
ask = "Continue from the last point and add one short paragraph."
url = "http://127.0.0.1:8082/v1/chat/completions"
def ask_model(messages, n=128):
    return requests.post(url, timeout=3600, json={
        "model": "local", "messages": messages, "max_tokens": n,
        "temperature": 0, "stream": False, "reasoning_effort": "none"})

r1 = ask_model([{"role": "user", "content": doc}])
print(r1.status_code, r1.json()["timings"])      # 200, prompt 185,598, ~130-190 s

r2 = ask_model([{"role": "user", "content": doc},
                {"role": "assistant", "content": r1.json()["choices"][0]["message"]["content"]},
                {"role": "user", "content": ask}])
print(r2.status_code, r2.text)                   # 503 "the engine stopped unexpectedly (exit code -6)"
```

After the 503 the engine is reloaded (~2–4 min on this box) and the next request that re-reads the prompt returns the #879 NaN instead. The full harness (13 requests, including the compaction turn) is `loadtest180k.py`; the JSON artifacts and the full engine log for every run above are available if you want them attached.

### Things I checked so I would not send you a red herring

* `--prefill auto:16384` → same failure (5/13). `--prefill auto` → same.
* `--adapt-every 0` (adaptive tier off) → same failure (5/13, 3 restarts). The tier is not it.
* `STRATA_PF_GEMM` on and off → same.
* `--layer-split auto` and `--layer-split 24` → both fail at long context (`auto` also produced #879 at position 39,613 on a 32k prompt).
* `--vram-reserve-mib 2048` → `3072`: no change for this failure (the known #267 mitigation is already in place).
* Client-side timeouts are not involved: the client waits 3600 s; the 503 bodies above are produced by the server.
* 8k / 32k paged conversations on the same config: every request returned 200 in this whole campaign — the failure is specific to *re-reading a long context*, not to the 2-GPU arrangement as such.

### What I think is happening (and what I am testing next)

The dying stage is literally named `verify window (pipelined)`, i.e. a stage that only exists while the prefill pipeline is on — so **`--pipeline-windows 2` is my first suspect**, possibly together with the VRAM budget at that context (`--kv-resident 32768` + `--conversation-cache-mib 8192` + `--vram-reserve-mib 3072` left 2,806 MiB free when the failing request began). This is a different stage from the decode-side pipeline stall fixed in v0.1.42 by #1741 (that one was the device switch in the decode poll loop; this one is the verify window at the tail of a long prompt), and it is on Linux/HIP, not Windows/CUDA.

I am running the single-variable version now — the shipping config with **only** `--pipeline-windows` changed to `0` — and will post the result as a comment. If it passes, that is the fix for my config; if it fails, I will bisect the remaining knobs and report which one it is.

### Related

* #267 — the verify-window watchdog this fires (already mitigated here with `--vram-reserve-mib 3072`, which is not enough at 185k with the pipeline on)
* #2449 — the same watchdog on long-context decode
* #2452 — #879 non-finite logits at a fixed prompt position
* #879 — the non-finite logits tracker
* #1740 / #1741 — the pipelined **decode** stall on Windows, fixed in v0.1.42 (different stage)
* #2637 — `--pipeline-windows 2` needing `STRATA_MTP_STAGE0=1` on gfx1100 (set here)