#5562 [Engine] prefill: the fused layout's shrink asks every layer's path, not only the formats
open · draft · @lwYeo · 0 评论 · 去 GitHub 看
BenchmarksMulti-GPUAMD / HIPNVIDIA / CUDAModels & quantsWindows
说明
# [Engine] prefill: the fused layout's shrink asks every layer's path, not only the formats ## Problem and result On a layer split over unlike AMD cards, `STRATA_PF_FUSED=1 STRATA_PF_FUSED_Q50=1` on Unsloth UD-IQ4_XS-v31 aborted every prompt at the first Q5_0-down layer of the stage whose card covers the format: ``` prefill: the fused layout's MoE buffers are too small for layer 13's expert path at a chunk of 9216 tokens ``` Reproduced seven times (layers 13 and 15, chunks 8960-11008) on an RX 6800 XT (gfx1030) + AI PRO R9700 (gfx1201) split `0-12 / 13-47`; reproduced locally, not static analysis. With this change, same pack, same split, same flags: the fused kernels run (`prompt experts on the fused int8 kernels`), several prompts served including a cold 74,953-token prefill, and the string `too small` does not occur in the log. The abort was the backstop working as designed; the shrink decision that sized the buffers is what disagreed with the layer loop. ## Related discussion #3577 / #4380 report this error on NVIDIA splits and fixed `fused_ring()`'s per-device cache - the same rule: the MoE buffers shrink only when every layer that runs in them can take the fused path. That fix covers the format question; this PR applies the same rule to the other half of the decision, the global `mmq_plan()`, and mirrors the layer loop's remaining static conditions. #5296 reports the same pack's Q5_0-down layers leaving the fused prompt path on a single gfx1151 card - the same formats this PR's plan entry covers. Searched issues and PRs for the error string: this RDNA2 + RDNA4 case is not reported. ## The problem in the code Two decisions disagree: - `fused_ring()` is per-device (the unlike-cards fix, `prefill.cpp`): on the RDNA4 stage every format in the pack is covered, so `fused_layout()` shrinks the MoE buffers to `stream_all_min() - 1` tokens. - `mmq_plan()` is one global, decided on whichever device opened the model first - here the RDNA2 stage, where `fused::native_supported()` is false for the Q5_0 down projections. Its fused-only entry `fo[]` baked that per-device answer into the global plan, so on the RDNA4 stage those layers had neither `layer[]` (no MMQ tile anywhere) nor `fo[]`: the layer loop sent them to the dequantize + FP16 GEMM at the full chunk, writing `T*K` rows into the shrunk buffers - the backstop's case. Reproduce: UD-IQ4_XS-v31, `--layer-split auto` (chose `0-12 / 13-47`), `STRATA_PF_FUSED=1 STRATA_PF_FUSED_Q50=1`, any prompt above ~2K tokens. ## Approach 1. **`mmq_plan()` records only the device-independent half.** A no-MMQ-tile layer of a native pack gets `fo[l] = 1` whenever a native fused kernel *can* take its formats; which device's kernels do take it is already asked per chunk by the `fused_nat` gate (`native_supported(mmq_gt, mmq_dt)` on the current device). The plan's manifest note says so. 2. **The shrink mirrors the layer loop's other static conditions** (`fused_shrink_ok()`, in `fused_layout()`): a layer with no planned path at all, a WMMA-tile layer and an APR-folded layer take MMQ or the FP16 GEMM at the full chunk, so the buffers shrink only when every layer is on the fused path - `fused_ring()` still covers the formats per device. Together the shrink and the per-layer decision derive from the same predicates, and the backstop stays what it was written to be: the drift guard, not the steady-state answer. Net on the repro: the RDNA4 stage runs its Q5_0-down layers fused with the shrunk buffers; the RDNA2 stage declines the shrink (its formats are not covered there) and keeps MMQ's buffers for its fallback. Packs whose layers all had a planned path shrink exactly as before, and the layer loop already re-checked `native_supported()` per chunk, so configurations that worked keep the same paths. ## Validation Branch `feature/fused-shrink-split-safe` @ `e1b8a69b`, base main `146df3c4`; build `tools/hip/build_windows.bat` (ROCm 10.2.0a20260930, hipBLASLt 100500, engine 0.1.58, gfx1030/31, gfx1100/01/02, gfx1151, gfx1200/01). **Passed:** - HIP compile of `src/prefill/prefill.cpp` and the full engine link, all 8 archs. - The repro configuration (Unsloth UD-IQ4_XS-v31, RX 6800 XT + AI PRO R9700, `--layer-split auto` - the engine chose `layers 0-12 (CUDA0), 13-47 (CUDA1)`, `STRATA_PF_FUSED=1 STRATA_PF_FUSED_Q50=1`): the log shows `prompt experts on the fused int8 kernels`, several prompts served - including a cold 74,953-token prefill at 1,195 tok/s and warm reuse turns at 58-61 tok/s decode - with no buffer abort. The same configuration on the pre-fix build aborted at layer 13 on every attempt. **Failed:** none. **Not run:** CUDA and other backends (the change is backend-neutral C++ in `prefill.cpp`, but only the HIP build was compiled); the C++ unit tests (they need a GPU box of their own); a controlled speed comparison - on this box the workload is lend/SSD-bound at the measured chunk, which masks any fused gain (cold prefill 1,195 tok/s fused vs 1,252 tok/s with `STRATA_PF_FUSED=0`, different lend conditions, so not a like-for-like number). Hardware: AMD Ryzen 7 7700X, 48 GB RAM, Radeon RX 6800 XT 16 GB (gfx1030) + Radeon AI PRO R9700 32 GB (gfx1201), Windows 11. ## Limits - The change is backend-neutral C++ in `prefill.cpp`, but only the HIP backend was built and run here; CUDA and SYCL were not compiled. - No controlled speed comparison (see Not run): this PR makes the fused path *reachable* on such splits, it does not claim a measured gain - on the reporting box the workload is lend/SSD-bound at the measured chunk, which masks any fused effect. - `fused_shrink_ok()` reads the static conditions (planned path, WMMA tiles, APR folding); the chunk-level ones (`stream_all`, `no_peer`) were already mirrored by `fused_layout()` and `fused_ring()`. The backstop at the layer loop is kept as the drift guard for anything else.
本站相关内容
相关页面的快捷入口。