Pull requests / #2230
#2230 [HIP] Keep mixed-gfx prefill GEMM modes per device (#690, #703)
open · @PJGV333 · 0 comentários · No GitHub
Descrição
## Summary Long prefill on heterogeneous AMD GPU layer splits has reported `hipErrorNoBinaryForGpu` after execution switches devices. This change improves the diagnostics and adds a per-device prompt GEMM precision policy: - Preserve `STRATA_HIP_PROMPT_F16=1` as the historical all-device FP16 mode; `=0` keeps BF16 everywhere. - Add `STRATA_HIP_PROMPT_F16=gfx103x` to select FP16 only on gfx103x (RDNA2), leaving gfx1201 on BF16. This is the mixed-device workaround validated here. - Add FP16/BF16 non-finite encoding checks, nontrivial GEMM tests against a CPU double-precision reference, and HIP BLAS/kernel diagnostics. - Make the mixed-prefill runner default to `gfx103x`, allow selecting all three modes, and remove `--vision` plus its optional value from text-only temporary configs. ## Validation Validated on an RX 9070 XT (`gfx1201`) + RX 6750 XT (`gfx1031`) with ROCm HIP 7.2.53211. Dispatch and per-device GEMM-reference tests pass for `=0`, `=1`, and `=gfx103x`; the mixed cross-device GEMM/GR return sequence was validated with `=gfx103x`. CTest passed 10/10 targeted tests and the runner's Python tests passed 3/3. With the updated patch, six text requests and same-conversation follow-ups at 16K, 49K, and 80K completed with PLE/MTP enabled and no HIP BLAS or kernel-image error. A separate multimodal run of the updated patch completed a 49K image+text prompt and follow-up using a generated 128x128 synthetic image; it does not reproduce the original image or prompt. An earlier test of the first functional version used a real image: 49K prefill, 43,478 generated tokens at 63.5 tokens/s, and a 93K-token follow-up. That real-image result is from the earlier version, not the updated patch's synthetic-image run. The generated GEMM reference checks use relative-L2 limits of `<1e-4` for BF16-in/FP32-out and `<2e-3` for FP16 output, plus maximum absolute error `<5e-3`. These bounded matrix checks do not establish bitwise identity, full numerical equivalence, or equal model quality between FP16 and BF16. ## Scope and limitations The workaround is validated on the gfx1201 + gfx1031 pair and ROCm HIP 7.2.53211 only. Do not interpret this as a universal fix for issue #690. The observed error occurs at a hipBLAS GEMM boundary; the proposed rocBLAS kernel/code-object or runtime-state explanation remains a hypothesis, not an isolated root cause. gfx1100 + gfx1030 and other ROCm versions or mixed pairs still need validation.
No site
Links install, modelos, releases.