Pull requests / #3575

#3575 [Engine] MXFP4 routed experts (MXFP4_MOE GGUFs) and MXFP4 PLE tables

open · @VLOD-ZDOV · 0 comments · View on GitHub

Setup & installAMD / HIPNVIDIA / CUDAModels & quantsWindowsLinux

Description

## Problem and result
GGUFs made with llama-quantize's `MXFP4_MOE` keep every routed expert (gate, up and down) and the PLE table in MXFP4
(ggml type 39). The engine refuses them at start: `layer 0's experts are MXFP4/MXFP4 (ggml types 39/39), which this
engine has no GPU kernels for`. With this change such a file runs as a native pack (`tools/iq_pack.py --compat-bf16`,
unchanged) and answers correctly. Files in the formats that already run take the same kernels as before.

## Related discussion
Replaces #1724, which was closed in the tracker reset and could not be reopened after the rebase on current `main`. In #1724 the maintainer wrote that format support
was outside that speed pass and that the 16-lane fix inside it looked worth a separate look. If format support is still
out of scope, I can split the 16-lane fix into its own pull request instead.

## Approach
- `src/kernels/cuda/iq_kernels.cu`: MXFP4 as an expert format for gate/up and down (`Fmt<39>`, `Split<39>`), next to
  NVFP4. It is IQ4_NL's layout with the FP4 table, one E8M0 exponent per 32 values and 17-byte blocks, after llama.cpp's
  `vec_dot_mxfp4_q8_1`. The prompt path dequantizes MXFP4 (`dq_mxfp4`) and takes the existing FP16 fallback.
- **The 16-lane fix:** `native_gu_multi_kernel` took its 16-lane path (`row_dot_80_sub16`, 80 dot calls per row)
  whenever `n_embd == 2560`. That held for every gate/up format so far (256-value blocks: 10 x 8 = 80), but an MXFP4 row
  is 160 calls, so only half of each gate/up row was computed. The kernel and the host's grid now test the row length
  itself (`nb * ipb == 80`). For the existing formats the condition is the same.
- `include/strata/artifact/dequant.hpp`: `dequantize_mxfp4`, after ggml's `dequantize_row_mxfp4`.
- `include/strata/kernels/ngram.hpp`, `src/kernels/ngram.cpp`: MXFP4 as a PLE table format.
- Tests: an MXFP4 fixture in `iq_parity` / `tools/iq_fixture.py`, MXFP4 in `native_grouped_parity` and
  `ple_reader_test`, and `check_old` in `native_grouped_parity`, which compares the default kernels with the one-row
  kernels at the model's shapes (2560 x 640). The v1 comparison could not catch the bug above, because both sides take
  the same gate/up kernel.

Compatibility: opt-in by file format only. No default, flag or setting changes.

## Validation
**Passed** (CUDA build, Linux, revision b23f9851 on top of `main` cbfd5b92):
- `iq_parity`: 0 failures. MXFP4 dequant rel 0.00e+00 against gguf-py, MMVQ rel 5.48e-03 (the other formats are in the
  same range), multi-column calls bitwise equal to one-column calls.
- `native_grouped_parity`: 0 failures. MXFP4 with every format pair bitwise equal to v1; `check_old` at 2560 x 640
  rel 0.00e+00 against the one-row kernels. With the old `n_embd == 2560` condition put back, `check_old` fails MXFP4
  (rel 8.7e-01), so the check covers the bug.
- `ple_reader_test`: every format including MXFP4 OK. `gguf_reader_test`: all passed.
- Default path unchanged: a UD-Q4_K_XL GGUF of Qwen3.8-Flash-Next served by this branch and by `main` built from the
  same tree, three greedy prompts of 200 tokens each: all three answers byte-identical (md5 of the text), and `main`
  run again before and after gives the same three.
- A real MXFP4_MOE GGUF of Qwen3.8-Flash-Next answers correctly and coherently. Before the 16-lane fix it looped on the
  same paragraph.

**Failed:** none observed.

**Not run:** HIP and SYCL builds (`iq_kernels.cu` is also compiled for HIP), Windows, other GPU generations, Mac.

## Limits
- The prompt path's MMQ has no MXFP4 kernel, so MXFP4 layers use the FP16 fallback there (slower prompt processing
  than the native formats).
- `setup.py` still lists MXFP4 files as unsupported; left for a separate change.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Related on this site

Quick links to related pages.