Issues / #3431

#3431 [Engine] Windows HIP (gfx1100): batched PLE block silently misreads prompts in ~1 of 8 engine starts; STRATA_PLE_BATCH=0 fixes it

open · @giacomobord-prog · 0 コメント · GitHub で見る

BenchmarksSetup & installAMD / HIPModels & quantsWindows

本文

**Setup**
- Windows 11, RX 7900 XTX (gfx1100) + RX 7900 XT, i9-13900K, 64 GB DDR5; AMD Adrenalin 26.9.2 (driver 32.0.32015.2008)
- Engine 0.1.45.3 built locally with `tools\hip\build_windows.bat` (`STRATA_HIP_ARCHS=gfx1100`, `STRATA_NO_VISION=1`), ROCm 10.2.0a20260930 (TheRock wheels)
- Qwen3.8-Flash-Next GSQ-RCO IQ3_S, one card (XTX, `"gpu": 0`), 64K context, `--kv int8`, MTP `--spec 4`, `--prefill auto`, no hipBLASLt tuning table

**Symptom**
In some engine starts every prompt longer than ~64 tokens is misread for the whole session: no error, no NaN, the model answers fluently but about something else ("your message is just `##`", answers in Chinese, an invented question). Prompts of 64 tokens or fewer are always read correctly. Restarting the engine usually clears it. The log of a bad start is identical to a good one (same slots, free VRAM, chunk size); the only visible difference is that a 176-token prompt is read ~15% faster in bad sessions (1.39-1.44 s vs 1.62-1.70 s).

**Reproduction**
A script that starts the server, sends two prompts (102 and 176 tokens, answer must contain a given word/number) and stops it, repeated:

| Configuration | Bad starts |
|---|---|
| default | 3 / 24 |
| `STRATA_PF_STEP_SYNC=1` | 2 / 16 |
| `STRATA_WMMA_GEMM=1` | 1 / 24 |
| `STRATA_PLE_BATCH=0` | **0 / 24** |

So the batched PLE block at layer 1 (`prefill.cpp`, `if (l == 1 && ple_on && ple_batch)`: the BF16 key/value GEMMs + `native_ple_postops_batch`) is involved. It does not look like a race between prefill steps (step sync does not help) nor hipBLAS alone (WMMA GEMMs still failed once). The bad state seems to be decided once per engine start and then stays for the session. Possibly related: #224 (illegal memory access in the same block) and #541 (HIP stall avoided by `STRATA_PLE_BATCH=0`), both closed - here it fails silently instead.

**Workaround**
`"env": {"STRATA_PLE_BATCH": "0"}` in the run config. Cost on this machine: prompt 810 -> 710 tok/s at 3.5K tokens, 865 -> 814 tok/s at 10.7K; decode unchanged (52-70 tok/s).

Logs of failing starts and the test script are available if useful.

関連リンク

インストール・モデル・リリースへのサイト内リンク。