#6433 [Engine] Fix: read the MTP draft layer's expert count from experts.bin, to support models with other expert counts such as pruned models

open · @ByteJuggler · 0 Kommentare · Auf GitHub

BenchmarksAMD / HIPNVIDIA / CUDAModels & quantsDocumentationWindowsLinux

Beschreibung

## Problem and result

The current code does not support MTP heads with a different number of routed experts than the default 512. The engine always builds the
MTP draft layer for 512 experts, whatever the `--mtp` files hold, so a head with any other count fails to load
(`mtp: experts.bin is truncated`). The tool that writes those files, `terminal/tools/mtp_rt.py`, also assumes 512.
And Strata cannot read an MTP head that ships only as llama.cpp's standalone draft GGUF, the form such heads are
often published in.

With this change, an experimental 256-expert head that previously failed now loads successfully. The original
standard 512-expert head is unchanged: same code, byte-identical runtime files, and the same acceptance as the
official (0.1.63) engine we tested against.

The specific reason for this change now: we have been testing and evaluating a pruned variant of Qwen3.8-Flash-Next in
Strata. Because it is a Flash-Next derivative, we think it may interest other Strata users, and we may open a PR later
to add support for it. That isn't possible without this fix. Even so, the change itself is general and fully
backwards compatible, and it also makes future experiments with other models easier. It is not only about this one
model.

## Approach

1. **Engine** (`engine/src/program/generate.cpp`). `draft_geometry` is no longer `const`. Before the draft layer loads,
   the engine reads the size of `<mtp>/experts.bin`. If it holds a whole number of expert blobs (`kernels::cpu::BLOB`)
   other than the compiled count, `draft_geometry.n_expert` takes that number. When it does, the engine logs the count once at start-up ("mtp: experts.bin holds 256 routed experts"), otherwise it logs nothing. Nothing
   changes when the file holds 512 blobs. A size that isn't whole blobs also changes nothing: it goes to the loader's
   own size check, as before. The MTP code already takes the count from `g.n_expert`. Its 512-only router fast paths
   fall back to the general `router_top10`, as the trunk's do for a pruned model.
2. **`terminal/tools/mtp_rt.py`** reads the count from the packed GGUF's `gate_up_proj` shape instead of fixing it at
   512.
3. **`terminal/tools/mtp_from_gguf.py`** (new) undoes each change llama.cpp's Qwen4Exp converter makes, giving the
   BF16 `mtp.*` tensors in the folder layout `mtp_fetch.py` writes, so that `mtp_pack.py` and `mtp_rt.py` work
   unchanged:
   - names;
   - norms stored as 1 + w in F32;
   - `eh_proj`, split in two;
   - `index_qk_proj` and `gate_up_proj`, joined again;
   - the newer merged `hc_*_down_inject` layout (llama.cpp #31028, #32102), split again.

   It stops on any tensor it doesn't know. `--verify` compares every tensor's bytes with an `mtp_fetch.py` folder.
4. **`terminal/tools/test_mtp_from_gguf.py`** (new) runs without model files.

## Validation

Two machines, both Windows 11:
- **NVIDIA:** Ryzen 5 5600X (AVX2), 64 GB DDR4-3200, RTX 3090 24 GB (262.5 W limit). Build: MSVC with CUDA 13.0.2,
  `STRATA_ENABLE_CUDA=ON STRATA_PORTABLE=ON STRATA_Q6K_EXPERTS=ON CMAKE_CUDA_ARCHITECTURES=86`, target `strata`.
- **AMD:** i7-5930K (AVX2), 192 GB DDR4-2400, RX 6900 XT 16 GB (gfx1030), AMD Software 26.11.2 (driver
  32.0.21045.29003). Build: `terminal\tools\hip\build_windows.bat` with `STRATA_HIP_ARCHS=gfx1030 STRATA_NO_VISION=1`,
  TheRock ROCm 10.2.0a20260930 (the official package's version).

Speed check on both: `ab_engine.py`-style A/B, base against `--mtp`, 2 alternating rounds, greedy, 3 chats and one
6,345-token prompt. The "before" and "official" runs use the release's own engine packages (0.1.63 Windows CUDA and
Windows HIP `rdna2`, SHA-256 checked against `SHA256SUMS.txt`). Acceptance differs between the machines because
their configs use different `--spec-min-p` values; each comparison below uses the same config on both arms.
With `--spec-min-p 0.9` on both machines, the 256-expert head's acceptance on HIP was 86% (This was on a different,
longer workload at 216K context) against 84% on CUDA.

**Passed, NVIDIA (CUDA)**
- **Build** of the branch (sm_86, the flags above).
- **256-expert head** (a pruned Qwen3.8-Flash-Next derivative, Q4_K_M):
  - Before: the official 0.1.63 Windows CUDA engine refuses it at start-up, `mtp: experts.bin is truncated`
    (exit code 1).
  - With this branch, the same command and files load it, log "experts.bin holds 256 routed experts" and start up
    normally (exit code 0).
  - Drafts accepted: 345 of 409 (84.4%) and 348 of 416 (83.7%).
  - Decode medians without / with MTP: 49.4 / 54.6, 53.6 / 56.9 and 55.5 / 62.0 tok/s on the three chats, and
    53.7 / 56.0 on the long prompt.
- **512-expert head** (Qwen3.8-Flash-Next IQ3_S, its own head):
  - It takes the old path: no expert-count line is logged.
  - Drafts accepted with this branch: 481 of 630 (76.3%) in both rounds.
  - On the official 0.1.63 Windows CUDA engine, same config and prompts: 76.0% to 76.3% over 6 rounds.

**Passed, AMD (HIP)**
- **Build** of the branch (gfx1030, the flags above).
- **256-expert head** (the same files: every SHA-256 matched the NVIDIA machine's):
  - Before: the official 0.1.63 Windows HIP `rdna2` engine refuses it, `mtp: experts.bin is truncated` (exit code 1).
  - With this branch, the same command loads it ("experts.bin holds 256 routed experts", draft layer 496 MiB) and
    starts up normally (exit code 0).
  - Drafts accepted: 453 of 730 (62.1%) and 439 of 715 (61.4%).
  - Decode medians without / with MTP: 24.0 / 27.6, 27.6 / 28.8 and 26.1 / 27.6 tok/s on the three chats, and
    25.1 / 28.7 on the long prompt.
- **512-expert head** (Qwen3.8-Flash-Next IQ3_S, its own head):
  - It takes the old path: no expert-count line is logged.
  - Drafts accepted with this branch: 516 of 797 (64.7%) and 492 of 772 (63.7%). On the official engine: 502 of 797
    (63.0%) and 502 of 807 (62.2%).
  - Decode without MTP is the same on both engines: 26.6, 30.6 and 29.8 tok/s on the chats; 28.9 against 28.4 on the
    long prompt.
- **`mtp_rt.py`** and the **unit tests**: the same results as below (byte-identical files, 26 tests OK).

**Passed, the official Windows HIP `rdna2` package on a real card**

`docs/AMD_HIP.md` lists this package (RX 6000: gfx1030, gfx1031, gfx1032) as built but not yet tested on a real card.
On the RX 6900 XT above, the unchanged release package:
- `strata-device.exe --list-devices` (with the package's `rocm\bin` on PATH): `device 0: AMD Radeon RX 6900 XT, arch
  gfx1030, 16.0 GiB, wave32`.
- `strata-device.exe --selftest`: `strata-device selftest OK` (compiled for gfx1030, gfx1031, gfx1032; 40 CUs; the
  package's own `amdhip64_7.dll`).
- Qwen3.8-Flash-Next IQ3_S with its own MTP head: 16 requests over 2 rounds, no errors. First answer: 256 tokens at
  34.6 tok/s, 179 of 212 drafts accepted. Long prompt: 6,345 tokens read at 942 tok/s, then 31.7 tok/s.
- No workaround was needed beyond `rocm\bin` on PATH for `strata-device.exe`, as the docs say. Only gfx1030 was tested.

**Passed, both machines**
- **`mtp_rt.py`:** the new version writes byte-identical `experts.bin`, `dense.bin` and `dense.txt` for both heads,
  against the files the runs above used. For the 512 head those files were made by the unchanged tool.
- **`mtp_from_gguf.py --verify`:**
  - Round trip: Qwen3.8-Flash-Next's 31 BF16 MTP tensors went through llama.cpp b38960f `convert_hf_to_gguf.py --mtp`
    and back through this tool.
  - 29 tensors are byte-identical to the originals.
  - The other two are norms: 1 value in 10,240 differs by about 3e-8, where F32's 1 + w had already lost the bits
    (`|w|` below about 1e-5).
- **Unit tests:** `python -m unittest discover -s terminal/tools -p "test_mtp_*.py"`: 26 tests, OK, including the 4
  new ones:
  - split and merged layouts round-trip to the original bits;
  - a non-512 head keeps its count;
  - an unknown tensor stops the tool.

  A mutant that skips the norm undo fails 2 of them.

**Failed**
- None, on either machine.

**Not run**
- **Builds:** SYCL, Metal and Linux (CUDA and HIP). HIP for gfx1031 and gfx1032.
- **CTest:** no CTest or `--selftest` covers this block. The rule isn't split out into a testable function.
- **Quality of the drafted output:** greedy text with MTP on differs run to run by design, so it isn't a check here.
  The check was acceptance and speed only.
- **Batch slots and layer splits** with a non-512 head, and a non-512 head above 64K context on HIP.

## Limits

- Only the expert count is taken from the files. Any other way a head differs from the canonical geometry still
  needs the compiled values. Hidden size, expert width and the hyper-connection count are all checked by
  `mtp_rt.py`'s shape assertions.
- `mtp_from_gguf.py` reads BF16/F32 draft GGUFs only. A quantized draft GGUF can't be restored exactly, so it is
  refused.
- Two machines (one NVIDIA sm_86, one AMD gfx1030), both on Windows, and one pruned model.

Mehr auf der Site

Links zu Install, Modellen, Releases.