Issues / #3170

#3170 Windows HIP gfx1100 (7900 XTX): silent engine death with --prefill auto:32768 at --max-context 204800 on 0.1.45 — larger chunk, no WER, intermittent (#1630 follow-up)

open · @yahosj-art · 1 commentaires · Sur GitHub

BenchmarksAMD / HIPModels & quantsSecurityWindows

Description

Follow-up to #1630 (closed). Same card family (RX 7900 XTX, gfx1100, Windows), newer engine,
different corner of the chunk × context matrix, and a different death mode.

## What happened
Engine 0.1.45, serving normally for ~80 min (large cold prefills at 1300–1500 tok/s, expert hit
87–90%, KV 99.9%, 0 file fallbacks). Mid-session it died between two requests: the log ends
right after a completed request line, no error, no exit message.

Death mode is **silent**: no `unspecified launch failure` line in the log, no WER report, no
Application Error event, no crash dump, no LiveKernelEvent/TDR, no commit spike. The process is
just gone (port released). This differs from #1630, where the fault line was logged and WER later
cloned the dead process (+69 GB commit).

## Environment
- Windows 11 (80 GB RAM, page file 42 GB at the time → commit limit 121.9 GB)
- RX 7900 XTX (gfx1100), PCIe 3.0 x16, Adrenalin current branch
- Strata engine 0.1.45 (prebuilt HIP), hipBLASLt tuning gfx1100-100500
- Model: Qwen3.8-Flash-Next IQ3_XXS pack (experts 45.3 GiB, resident RAM 39.11 GiB)
- Args (relevant): `--prefill auto:32768 --max-context 204800 --kv int8 --kv-resident 32768
  --resident-experts --spec 4 --mtp rt --vram-reserve-mib 1024`
- env: STRATA_PF_FUSED=1 STRATA_PF_GEMM=1 STRATA_DENSE_MMQ=1

## Frequency (this config, 8 days)
- Dozens of large cold prefills through the 32768-token chunk path: OK (1465–1500 tok/s).
- 1 death in 8 days, mid-prefill, silent.
So at chunk 32768 × ctx 204800 the fault is **intermittent**, not 3/3 as #1630 saw at
8192 × 262144.

## Data points vs #1630
| chunk | ctx | engine | result |
|---|---|---|---|
| 8192 | 262144 | 0.1.41–0.1.43 | 3/3 fail, logged launch failure, WER +69 GB |
| 8192 | 131072 / 524288 | 0.1.42 | OK |
| 32768 | 204800 | 0.1.45 | ~dozens OK, 1 silent death in 8 days (no WER, no fault line) |
| 7500 | 204800 | 0.1.45 | not yet run (we moved to this as the guard) |

## Questions
1. Does the fault depend on the chunk × context interaction (non-monotonic: 262K fails, 512K OK)?
   What does the 32768 chunk change in the prompt path vs 8192 (ring depth, pinned buffers)?
2. Can the fault ever be silent (no `prefill copy_i32` line, no WER)? Is the crash handler
   expected to log something on device loss?
3. Any kernel switch (STRATA_PF_* / hipBLASLt table) known to avoid the failing launch at large
   chunks, so users can keep fast prefill?
4. Would you accept a repro effort from a user with the card? We can run a chunk × context matrix
   (e.g. 7500/8192/16384/32768 × 131072/204800/262144) with commit/RAM sampling.

## Mitigations we applied (for the record)
- `--prefill auto:7500` (per #1630 measurements), auto-restart wrapper, page file fixed 80 GB.
- Considering `--max-context 131072` + `--prefill auto:32768` to keep prefill speed inside the
  context zone that #1630 measured clean — is that a reasonable guess or does the danger zone
  move with the chunk?

<sub>This report was drafted entirely on top of the engine it describes — Strata 0.1.45 serving
Qwen3.8-Flash-Next IQ3_XXS as the author's daily driver. The crash cost us a live session, which
is why the matrix offer above is serious.</sub>

Sur le site

Liens install, modèles, releases.