Pull requests / #2018

#2018 prefill: VGPR-capped gfx11 fused expert kernels back on for gfx1151 on clang 23 (same bits, 2-15% less time per layer)

open · @libratechw · 0 commentaires · Sur GitHub

BenchmarksAMD / HIPNVIDIA / CUDAModels & quantsDocumentation

Description

## Title
Issue: Refs #1180


## Summary

Since 0.1.40.2 the `waves_per_eu(8)` variant of `native_w11_kernel` is off on clang 22 and newer, because clang 22
computes wrong results with it on gfx1100 (#1180). On gfx1151 with clang 23 (ROCm 7.14.1) the capped kernels give the same
bits and take less time, as measured in #1180. This applies the cap there again, per format, and leaves every other
architecture and compiler as it is.

## What changed

`src/prefill/moe_fused_iq.cu` only.

- `NW_LB` is the variant's template argument again (8, or `-DSTRATA_W_LB=N`), the same in the host and every device pass.
- `nw_capped(WT)` decides in the device pass whether the cap applies: always before clang 22 or with `-DSTRATA_W_LB`; for
  gfx1151 on clang 23 for every format except IQ2_XS (gate/up spills 104 VGPRs under the cap and takes 61-66% more time) and
  Q8_0 (down, 18 spilled, -1.9% to +1.9%); otherwise off (clang 22 and 24+, and every other architecture).
- `hip_prefill_fused_iq --no-ref --only=<pair> --chunks=2048,3584,8192` (gfx1151, ROCm 7.14.1, clang 23), one layer, top 10 over
  512 experts, `fused` ms, median of 3 runs (range in brackets), the output hash is the same in every case. "Before" is
  v0.1.43 (uncapped); IQ2_XS gate/up stays uncapped, so its change comes from the capped IQ4_NL down:

| gate/up / down | tokens | before | after | change |
|---|---:|---:|---:|---:|
| IQ2_XXS / Q2_0 | 2048 | 13.73 (13.64-13.76) | 12.11 (12.10-12.13) | -11.8% |
| | 3584 | 21.17 (21.00-21.25) | 18.42 (18.35-18.45) | -13.0% |
| | 8192 | 40.57 (40.23-40.69) | 34.79 (34.69-34.84) | -14.2% |
| IQ3_XXS / IQ4_NL | 2048 | 14.31 (14.28-14.47) | 12.90 (12.87-12.90) | -9.9% |
| | 3584 | 22.57 (22.45-22.63) | 19.54 (19.53-19.59) | -13.5% |
| | 8192 | 42.28 (42.02-42.32) | 36.09 (36.09-36.40) | -14.7% |
| IQ3_S / IQ4_NL | 2048 | 14.66 (14.51-14.68) | 13.31 (13.27-13.32) | -9.2% |
| | 3584 | 22.98 (22.89-23.00) | 20.11 (20.06-20.12) | -12.5% |
| | 8192 | 42.80 (42.67-42.93) | 37.40 (37.26-37.40) | -12.6% |
| IQ4_XS / Q2_0 | 2048 | 14.67 (14.67-14.69) | 12.90 (12.90-12.92) | -12.0% |
| | 3584 | 21.96 (21.92-22.12) | 19.53 (19.50-19.54) | -11.1% |
| | 8192 | 41.48 (41.36-41.61) | 36.63 (36.39-36.68) | -11.7% |
| IQ2_S / Q2_0 | 2048 | 15.14 (14.85-15.20) | 14.74 (14.69-14.76) | -2.7% |
| | 3584 | 23.87 (23.64-23.89) | 23.10 (23.05-23.26) | -3.2% |
| | 8192 | 44.57 (44.20-44.62) | 43.01 (42.89-43.17) | -3.5% |
| IQ2_XS / IQ4_NL | 2048 | 15.11 (15.01-15.23) | 14.86 (14.85-14.88) | -1.7% |
| | 3584 | 24.01 (23.95-24.05) | 23.00 (22.91-23.04) | -4.2% |
| | 8192 | 44.84 (44.55-44.90) | 42.89 (42.80-42.98) | -4.3% |

- End to end, Ryzen AI Max+ 395 (gfx1151), UD-IQ4_XS, the section 5 switches of `docs/STRIX_HALO.md`, 3 interleaved rounds of
  fresh servers (tok/s, prompt):

| prompt | before | after | change | per round |
|---|---:|---:|---:|---|
| 4K | 868.0 | 894.9 | +3.1% | within the spread of 3 runs |
| 32K | 1,182.9 | 1,207.0 | +2.0% | within the spread of 3 runs |
| 128K | 1,126.1 | 1,153.4 | +2.4% | +1.8 / +2.4 / +2.6% |

  Decode unchanged; the same text for all 9 prompts.

What it leaves alone: the gfx1100 disassembly is identical to before (kernel names aside); gfx12; `-DSTRATA_W_LB`; clang 22
and 24+ keep the clang 22 rule. The path is opt-in (`STRATA_PF_FUSED=1`), so the default output is unchanged. The file builds
for gfx1012 / 1030 / 1100 / 1150 / 1151 / 1200 / 1201 and with `STRATA_HIP_GFX906`; not built with CUDA or SYCL (the changed
lines are inside `#if defined(__HIPCC__)`; SYCL has its own copy of this file).

## Extra Notes

Tested: `hip_prefill_fused_iq` passes on gfx1151 (ROCm 7.14.1, clang 23) after rebasing on v0.1.43.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Sur le site

Liens install, modèles, releases.