Pull requests / #2020

#2020 prefill: gfx11 F16 matrix-core routed experts (opt-in STRATA_PF_F16=1): UD-Q4_K_XL 32K +15.6%, UD-IQ4_XS 32K +11%

open · @libratechw · 0 comentarios · En GitHub

BenchmarksAMD / HIPNVIDIA / CUDAModels & quantsDocumentation

Descripción

## Title
Issue: Refs #1258 (where outside help pays on gfx1151)


## Summary
The routed experts were 6/10 of the prompt-time gap to gufo on gfx1151. `moe_f16_kq.cu` adapts gufo's paired
`RoutedF16GEMMKernel` (97b8645f, MIT, attribution and licence text at the top of the file) to Strata's tables, int8 activations
and H; `native_f16_kernel` (moe_fused_iq.cu, shared parts in `moe_f16_common.cuh`) does the IQ gate/up formats and the
Q2_0 / IQ4_NL / Q8_0 down. Codes stay packed in LDS, each wave turns its own rows into F16, the F16 WMMA accumulates all of K in
FP32. Off by default (the output differs from the int8 kernels'); gfx12 keeps the int8 kernels.

## What changed
Kernels as above, `STRATA_PF_F16=1` (K-quants also need `STRATA_PF_FUSED_KQ=1`; Q8_0 down needs neither), `docs/STRIX_HALO.md`,
tests: `hip_prefill_fused_iq` thresholds 5e-3 under F16 and a second ctest entry running it with the switch.
Measured on a Ryzen AI Max+ 395 (gfx1151, ROCm 7.14.1, clang 23):

| One layer, `hip_prefill_fused_iq` (16,384 tokens, top 10 over 512 experts) | int8 kernels | F16 | change |
|---|---:|---:|---:|
| Q4_K / Q5_1 (UD-Q4_K_XL), ms | 85.3 | 53.5 | -37% |
| IQ3_S gate/up, ms per 128-expert launch | 11.65 | 9.87 | -15% |
| IQ4_NL down, ms per 128-expert launch | 6.11 | 4.71 | -23% |

| End to end, prompt tok/s, 3 interleaved rounds, medians | before | F16 | change | gufo |
|---|---:|---:|---:|---:|
| UD-Q4_K_XL 32K (vs v0.1.41) | 1,235.7 | 1,427.3 | +15.6% | 1,512 |
| UD-Q4_K_XL 128K (vs v0.1.41) | 1,179.0 | 1,364.9 | +15.8% | 1,410 |
| UD-IQ4_XS 4K (vs v0.1.42) | 926.6 | 1,054.8 | +13.8% | |
| UD-IQ4_XS 32K (vs v0.1.42) | 1,257.4 | 1,397.1 | +11.1% | |
| UD-IQ4_XS 128K (vs v0.1.42) | 1,202.0 | 1,333.6 | +10.9% | |

Decode is unchanged.

First-token KL against a run with no switches (12 natural-text prompts of 1.5K-24K tokens, top-1 the same in 12 of 12 for every
arm), mean / median:

| | section-5 switches only | + int8 fused experts | + F16 (this PR) |
|---|---:|---:|---:|
| UD-Q4_K_XL | 0.0039 / 0.0019 | 0.0031 / 0.0024 | 0.0068 / 0.0021 |
| UD-IQ4_XS | 0.0181 / 0.0121 | 0.0068 / 0.0032 | 0.0085 / 0.0036 |

One 1.5K prompt carries most of the UD-Q4_K_XL mean (0.049). Only 12 prompts and one model family; a larger set from you
would be welcome.

## Extra Notes
Not run: other GPUs, CUDA/SYCL builds. The first request after start can read ~1 s slower with F16 (3 of 5 servers; cause not found).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

En el sitio

Enlaces a install, modelos, releases.