#6582 [Engine] support native GGUF experts (Q4_0, IQ4_NL) in MTP drafter
open · @nmqx · 0 commentaires · Sur GitHub
BenchmarksMulti-GPUAMD / HIPNVIDIA / CUDAModels & quantsWindowsLinux
Description
## Problem and result `MtpDrafter` currently assumes all 512 routed draft experts are stored in the legacy S2 `Q2_0` layout (`675 MiB`), which has high quantization error (`~0.429` relative RMS) compared to the BF16 MTP checkpoint. Meanwhile, `tools/mtp_pack.py` already supports packing `q4_0` and `q8_0` MTP GGUFs, and the engine's `native_expert_grouped` kernels already run native GGUF experts on GPU. This PR adds end-to-end native GGUF routed expert support (`Q4_0`, `IQ4_NL`, and other formats supported by `native_expert_grouped`) to `tools/mtp_pack.py`, `tools/mtp_rt.py`, `MtpDrafter`, and `stage_reserve` while keeping 100% backward compatibility with existing `Q2_0` S2 `rt/` packs: - **Expert reconstruction error (first 8 experts vs BF16):** `Q2_0` `0.429` RMS → **`Q4_0` `0.089` RMS (4.8× lower)**, **`IQ4_NL` `0.078` RMS (5.5× lower)**. - **End-to-end 2× RTX 3090 decode (Flash-Next GSQ-RCO IQ3_S, `--pipeline-windows 2 --spec 3 --mtp-q4 head --layer-split 24`, temp=0):** - On `v0.2.16` (`TAIL_SKIP=7`): `rt-q4_0` improved prose `+10.8 tok/s` and code `+3.8 tok/s` (`+3.2–5.6 tok/s` 3-avg); `rt-iq4_nl` improved `dec@32K` by `+11.6 tok/s` (`142.3 tok/s`). - On `v0.2.19` (`TS=7` + `WS=6`): `rt-q4_0` reached `207.0 / 153.0 / 146.6` (`168.9` avg) vs stock `Q2_0` `197.8 / 142.5 / 153.1` (`164.5` avg). Every drafted token is still verified by the main model, so output quality is exact. ## Related discussion none (backward-compatible opt-in runtime format for `mtp_pack.py` / `mtp_rt.py` / `MtpDrafter`). ## Approach - **`terminal/tools/mtp_pack.py`:** Adds `iq4_nl` quantizer/dequantizer alongside `q2_0`/`q4_0`/`q8_0`. - **`terminal/tools/mtp_rt.py`:** For `Q2_0` GGUFs, writes the exact legacy S2 `experts.bin` with no `experts.txt` (100% unchanged). For native GGUFs (`Q4_0`, `IQ4_NL`, `Q8_0`, etc.), writes raw GGUF blocks in `NativeExpertLayout` order (`[gate | up | down]` per expert) plus a small `experts.txt` metadata header (`format native`, `gu_type`, `d_type`, `n_expert`, `n_embd`, `n_ff`, `blob_bytes`). - **`engine/include/strata/core/mtp.hpp` & `engine/src/core/mtp.cpp`:** Detects `experts.txt` (`format native`), validates geometry and `experts.bin` byte size against `native_expert_layout(...)`, allocates `native_q8_1_bytes` / `native_expert_scratch_bytes`, updates `experts_bytes()` (prompt room lend/reclaim), and dispatches `native_quantize_q8_1` + `native_expert_grouped` in `record_rest()` (leaving legacy S2 `moe_grouped_s2` untouched when `experts.txt` is absent). Adds `MtpDrafter::reserve_mib(rt_dir)` scaling the VRAM reserve above `kDrafterMib` (`1000 MiB`) by the extra expert bytes (`~1675 MiB` for `Q4_0`/`IQ4_NL`). - **`engine/src/kernels/cuda/iq_kernels.cu`:** Adds `X(20)` (`IQ4_NL`) to `STRATA_GU_FMTS` so `IQ4_NL` gate/up experts compile into `native_expert_grouped` (note `v0.2.19` already fixed `gu_sub16_rows` via `gu_calls() == 80`). - **`engine/include/strata/program/stage_reserve.hpp`, `engine/src/program/generate.cpp`, `engine/tests/program/stage_reserve_test.cpp`:** Threads `MtpDrafter::reserve_mib()` into `stage_reserve::Inputs::drafter_mib` so multi-GPU layer-split sizing reserves the exact extra VRAM for larger MTP packs, with unit tests. ## Validation **Passed:** - **Build & unit tests (`v0.2.19` `c6f6427d9` + commit):** Windows 11, CUDA 13.3.73, MSVC 19.50, Ninja, `-DCMAKE_CUDA_ARCHITECTURES=86 -DSTRATA_ENABLE_CUDA=ON -DSTRATA_Q6K_EXPERTS=ON -DSTRATA_BUILD_TESTS=ON`. Clean `[171/171]` link; `stage_reserve_test.exe` → **`864 cases, PASS`**. - **Pack generation & backward compat:** Packed `rt-q4_0` and `rt-iq4_nl` from BF16 MTP tensors (`1,350 MiB` experts, `1,512 MiB` total drafter VRAM) and verified legacy `rt/` (`Q2_0 S2`, `837 MiB` VRAM, no `experts.txt`) loads and runs identically on the same binary. - **End-to-end 2× RTX 3090 runs (Flash-Next IQ3_S, temp=0, 3 reps):** - `v0.2.19` stock `rt` (`Q2_0 S2`, `TS=0`): `197.8 / 142.5 / 153.1` (`164.5` avg, `0` VRAM warnings). - `v0.2.19` `rt-q4_0` (`WS=6`, `TS=7`, `fixdef`): `214.4 / 147.7 / 148.2` (`170.1` avg, `depth-avg 128.3`, `0` VRAM warnings). - `v0.2.16` matched A/B (`p23-q4_0-ws6` vs `rt` `Q2_0`): `233.8 / 158.9 / 160.7` (`184.5` avg, `+9.0%` over baseline); `rt-iq4_nl` `dec@32K = 142.3 tok/s`. **Failed:** - none observed. **Not run:** - HIP (`gfx906`/RDNA), SYCL, Metal, and Linux builds/runs. - `Q8_0` MTP pack end-to-end benchmark (code supports any `native_expert_supported` type, tested with `Q4_0` and `IQ4_NL`). ## Limits - `Q4_0` / `IQ4_NL` MTP experts use **`+675 MiB` more VRAM** (`1,512 MiB` vs `837 MiB`) on the drafter GPU. On tight-VRAM multi-GPU splits with `STRATA_ROUTE_TAIL_SKIP=0`, displacing ~330 expert cache slots on GPU1 can offset the draft-acceptance gain on prose/chat (`v0219-q4_0-ws0` `158.6` vs stock `Q2_0` `164.5` under `TS=0`, while code still rose `197.8 -> 203.2`). Works best when GPU1 has ample VRAM headroom or with `STRATA_ROUTE_TAIL_SKIP=7`. - Not a default change: users must explicitly pack and pass a native MTP directory via `--mtp`.
Sur le site
Liens install, modèles, releases.