反馈 / #2245

#2245 gfx1030 2-GPU `--pipeline-windows 2` + MTP: verify hang (#267) bisected to 6126f8d; the GPU's MEC stops servicing the drafter and stage-1 queues

open · @metzenwest · 0 评论 · 去 GitHub 看

Setup & installAMD / HIPModels & quantsWindows

说明

**Setup:** 2× Radeon Pro V620 (gfx1030), Strata v0.1.43.1, Qwen3.8-Flash-Next IQ3_S, `layer_split 25`,
`--spec 3 --spec-min-p 0.7 --mtp rt-en --pipeline-windows 2`, ROCm TheRock 7.14.0a20260612 (7.13.0a20260515 also
reproduces).

**Symptom:** "verify: the window never finished; its GPU waits were released but the GPU did not finish within 5 s
(#267)". GPU breadcrumbs stop on stage 1 between `waitA` (19) and `VRAM hits` (20); the GPU is busy but nothing
runs. About 30% of runs fail; with `STRATA_VERIFY_TRACE=1` nearly every first request fails.

**Bisect:** first bad commit **6126f8d** ("HIP gfx103x: PR #540's attention kernel with 8 cells per step and DPP
lane exchanges"). The 9-step bisect ran from 2bfac51 (the PR #859 merge, good) to e8ca9af (0.1.40.2, bad).

**What it is (details in the ROCm issue: https://github.com/ROCm/legacy-rocm-build/issues/6698):** at the hang, the drafter's queue has `attn_chunk_kernel_pre75<1,8,true>`
next and stage 1's queue has `native_down_multi_kernel` next. Both packets are valid, every earlier packet is
retired, there are no waves, and the MEC pipes owning those queues are idle. It looks like a GPU firmware-level
stall that this kernel triggers through timing, not a bug in the kernel's math: no out-of-bounds access, every loop
bounded, 0 scratch.

**Variants tested (`STRATA_VERIFY_TRACE=1`, 4-8 requests each):**

| Drafter attention kernel | Result |
|---|---|
| 8 cells + DPP (default on gfx103x) | hangs |
| 8 cells, no DPP | clean |
| 2 cells + DPP | clean |
| 1 cell, no DPP | clean |
| verifiers 8 cells + DPP, drafter on the default kernel | clean |
| `STRATA_ATTN_LANECELL=1` (bypasses pre75 entirely) | clean |
| no MTP | clean |

Also tried, and still hangs: MTP drafter stream at high priority (fewer hangs, not zero), `GPU_MAX_HW_QUEUES=6`, a
synchronous chained prompt read (single submitting thread), scratch-reclaim off, `amdgpu.cwsr_enable=0`, ROCm 7.13.

**Suggestion:** until AMD weighs in, on gfx103x layer splits with MTP and pipeline windows, either keep the drafter's
decode attention on the default kernel, or prefer `STRATA_ATTN_LANECELL=1`. Both avoided the hang in our tests and
both are faster than our previous v620-based build (with upstream's opt-in switches, same flags, 2 rounds):

| Change | Decode: code | Decode: 16K | Decode: 32K | Prompt: 16K | Prompt: 32K |
|---|---|---|---|---|---|
| Drafter on the default attention kernel (verifiers keep 8 cells + DPP) | +2.2% | +2.3% | +0.4% | +8.5% | +7.9% |
| `STRATA_ATTN_LANECELL=1` (bypasses pre75) | +1.4% | +1.2% | −0.4% | +2.8% | +1.8% |

本站相关内容

相关页面的快捷入口。