#6515 [Engine] gfx12: `nw_barrier()` falls back to `__syncthreads()`, so the fused prompt experts' stage writes can race (latent)

open · @gho-cl0de · 2 Kommentare · Auf GitHub

AMD / HIPNVIDIA / CUDAModels & quantsWindowsLinux

Beschreibung

## Problem

`nw_barrier()` in `engine/src/prefill/moe_fused_iq.cu` (v0.2.17, line 1782) writes the LDS wait out explicitly only on
gfx11 (`STRATA_NAT_W11`). Its own comment gives the reason: in WGP mode the compiler may drop the `s_waitcnt` before
`__syncthreads()`'s `s_barrier` (llvm/llvm-project#220357), so a read after the barrier can overtake another wave's LDS
store. On gfx12 the function is plain `__syncthreads()`.

gfx12 runs these kernels in WGP mode too. `native_w11_kernel` relies on `nw_barrier()` at the top of each K stage to make
the previous stage's weight writes (`put(raw, (s + 1) & 1)`) visible before the stage reads them; the comment at line 1968
says so ("the next stage's first barrier orders these writes: nw_barrier waits for them explicitly"), but on gfx12 it does
not wait explicitly.

## What we saw

With the unmodified v0.2.17 kernels we have **not** seen a wrong or non-deterministic result: dozens of prompt reads on an
RX 9070 XT gave the same GDN state hashes every time. The race showed up only after a change to the kernel that alters its
instruction schedule:

- The change: on gfx12, the IQ3_XXS / IQ3_S gate/up decode reads its codebook (`iq3xxs_grid` / `iq3s_grid`) from the device
  table instead of the LDS copy `sgrid` (the same values; a faster decode, fewer LDS lookups in flight).
- Result: non-deterministic prompt reads. 6 of 6 engine starts on the same 62.5K-token prompt gave different GDN state
  hashes (`STRATA_STATE_HASH_GDN=1`), first differing at a different layer in each start.
- The kernel alone is fine: `hip_prefill_fused_iq` gives the same results with and without the change, so the codebook
  reads themselves are correct.
- With `STRATA_Q_TWOBAR` (the second barrier after each stage's weight writes): deterministic, the reference hash, 3 of 3
  starts.
- With `ALIAS` off for those kernels (the H tile in its own LDS): deterministic, the reference hash, 3 of 3 starts.
- With the gfx11 form of `nw_barrier()` on gfx12 (below), no second barrier: deterministic, the reference hash, 12 of 12
  starts (with and without the codebook change).

So we think the gfx12 path has the hole the comment describes for gfx11, and the current schedule happens not to hit it. A
different compiler, a different kernel instance or a small change to the kernel could.

## Proposed change

The gfx11 barrier for gfx12, with gfx12's own LDS counter:

```cpp
#elif STRATA_NAT_W12
    __builtin_amdgcn_fence(__ATOMIC_RELEASE, "workgroup", "local");
    asm volatile("s_wait_dscnt 0x0" ::: "memory");
    __builtin_amdgcn_s_barrier();
    __builtin_amdgcn_fence(__ATOMIC_ACQUIRE, "workgroup", "local");
#else
```

Side effect: the LDS-only fences drop the `global_inv scope:SCOPE_SE` that `__syncthreads()` emits at every stage (the
IQ3_XXS gate/up instance: 7 -> 4 invalidates, the IQ4_NL down instance 5 -> 2).

Speed on the RX 9070 XT, IQ3_S pack, 62.5K-token prompt, 3 interleaved starts each (`STRATA_PREFILL_TIMING=1`):
- This barrier with the LDS codebook as today: the expert phase 5,105 -> 5,194 ms in one session and 5,105 -> 5,157 ms in
  another (the explicit wait stalls on the decode's LDS lookups in flight); first read 17,330 -> 17,488 ms and 17,367 ->
  17,368 ms.
- This barrier with the codebook read from the device table: the expert phase 5,105 -> 5,071 ms, first read 17,367 ->
  17,303 ms (-0.4%).
- A fences-only form (no explicit `s_wait_dscnt`) was 17,274 ms and also deterministic in 3 of 3 starts, but the gfx11
  code keeps the explicit wait on top of the fences, presumably because the LLVM issue is the compiler dropping waits, so
  we would not rely on the fences alone.

## Validation

Passed (RX 9070 XT, gfx1201, Linux, ROCm 7.14, HIP build of `engine/` with `-DCMAKE_HIP_ARCHITECTURES=gfx1201`):
- `ctest -R "hip_prefill_fused|native_grouped_parity|native_i8_parity|prefill_decode_golden|hip_prefill"`: 19 of 19
  (`hip_prefill_fused_iq`, `_iq_f16`, `_q36`, `_iq_q8d`, `_iq_q8gu`, `_iq_q6k`, `_iq_tq`, `_iq_tqh`, `native_i8_parity`,
  `native_grouped_parity`, ...; the four `*_gemm*` ones skip on this machine as before).
- The GDN state hash of a 62.5K-token IQ3_S read equals the unchanged kernel's in every start (21 starts over three
  sessions).

Not run:
- **CUDA, SYCL and Metal builds: not built and not tested. We only have AMD hardware (one RX 9070 XT), so we cannot build or
  run anything else.** The bug and the change are AMD-only: the new branch is `#elif STRATA_NAT_W12`, which only gfx12
  device code takes, so every other build compiles `nw_barrier()` exactly as before.
- gfx11 (Strix Halo, RX 7900): not touched by the change, not tested.
- Other gfx12 cards (R9700, RX 9060 XT): not tested.
- Windows: not tested.
- Other compilers: only the ROCm 7.14 toolchain (`/opt/rocm/core-7.14/llvm/bin/clang++`).

## Limits

- We did not reproduce a failure of the unmodified v0.2.17 kernel; the race needed the codebook change to show. What we
  say about the unmodified kernel being at risk is our reading of the code and of the `nw_barrier()` comment, plus the
  experiment above, not an observed failure.
- The static check we tried (a linear scan of the device assembly for `ds_store`s not drained before an `s_barrier_signal`)
  found nothing in either version, because the weight writes sit at the bottom of the stage loop and the barrier at its
  top; we did not do a control-flow-aware analysis.

Mehr auf der Site

Links zu Install, Modellen, Releases.