#6594 [Engine] HIP - avoid NaN by using per-expert codecs for mixed prefill layers

open · @kdrapel · 0 Kommentare · Auf GitHub

Multi-GPUAMD / HIPNVIDIA / CUDAModels & quantsWindowsLinux

Beschreibung

[Engine] Fix Inkling-2-Mini SQ-15 NaNs in HIP prefil. It was crashing on my RX 9060 XT 16GB HIP (gfx1200).
Error in console was "the model's output became non-finite (NaN) twice in a row"

## Problem and result
Inkling-2-Mini SQ-15 produced non-finite logits on an RX 9060 XT when a HIP prefill batch reached 64 tokens. The direct CLI emitted 0,0 for the 64-token reproducer, and the API returned HTTP 503 when it detected non-finite logits. With the fix, all 12,803,712 logits in the 64-token regression check were finite.

## Related discussion
None.

## Approach
The large-batch HIP MoE dispatcher classified mixed-codec layers as MX8 and sent their SF4 and S_K weights to the MX8-specialized kernels. Classify the gate/up and down projections independently. Select the MX8 specialization only when every expert in that projection uses MX8. Mixed projections use the existing kernel that reads each expert’s codec from its descriptor.

## Validation
Windows, Radeon RX 9060 XT (gfx1200, 16 GB), TheRock 7.14.1 / HIP 7.14.60850, Inkling-2-Mini SQ-15. Baseline c6f6427d9, fix 9c3e8e57d.

Passed:
- cmake --build build-hip --target strata-inkling passed in the local Windows HIP build environment.
- Ran the following on 9c3e8e57d from the engine directory, with $model set to the local SQ-15 file. The dump contained 64 rows × 200,058 logits. numpy.isfinite(values).all() passed across all 12,803,712 values.

  $ids = (101..164) -join ','
.\strata-inkling.exe --native $model --max-context 32768 --draft off `
  --prompt-ids $ids --dump-ref logits.bin
  
<img width="2075" height="718" alt="image" src="https://github.com/user-attachments/assets/a219d0f2-2d83-4fe2-acb3-d726f6673579" />

Failed:
- Baseline reproduction on c6f6427d9. The command below printed 0,0 with 64 input tokens. With 63 input tokens (101..163), it printed 198,28. No failures were observed in the build or logits check on the fixed revision.

  $ids = (101..164) -join ','
.\strata-inkling.exe --native $model --max-context 32768 --draft off `
  --prompt-ids $ids --gen 2

Not run:
- Linux HIP, other AMD targets, and the release-pinned TheRock toolchain. Local validation used the Windows gfx1200 build.
- CUDA, SYCL, and Metal. The change is confined to the Inkling HIP backend.

## Limits
Hardware validation covered one GPU and one model pack with all experts resident on the GPU. Performance and multi-GPU behavior were not measured.

Mehr auf der Site

Links zu Install, Modellen, Releases.