Pull requests / #1934

#1934 mtp_pack: Q2_0 scale search over 0.15 to 1.0 x max|w|, either sign

open · @gputier · 0 comentarios · En GitHub

Setup & installNVIDIA / CUDAModels & quants

Descripción

`tools/mtp_pack.py` picks each Q2_0 block's scale among 17 positive values between 0.5 and 1.0 times the block's max |w|. The levels are -1, 0, 1 and 2, so they are not symmetric, and a negative scale is often closer: it puts the level 2 on the side where the block's largest weights are. This PR searches 0.15 to 1.0 times max |w| for both signs. It keeps the previous 17 candidates in the set and rounds each scale to fp16 before measuring it, so no block ends up with a larger error than before. The file format does not change, and neither does the engine: the kernels already read a signed scale.

Relative RMS error of the packed draft experts, on synthetic blocks (`tools/test_mtp_pack.py`): 0.855 of the previous search on normal weights, 0.824 on Laplace, 0.833 on Student t3. The test also checks every block, and that the packed bytes decode, through `tools/gguf_writer.dequantize_q2_0`, to the values the search measured. A scale too large for fp16 now raises an error. One that rounds to 0 keeps the previous behaviour.

`expert_multi_test` now gives half of its Q2_0 scales a negative sign and compares the kernel with the scalar oracle. `s2_expert_grouped_parity` already drew signed scales. On an RTX 5090 with a Ryzen 9 9950X3D (AVX-512), both pass, and the whole ctest gives the same 106 of 107 as v0.1.42.

On that machine (IQ3_S, a French draft vocabulary of 46,587 tokens, served with `--spec 4 --spec-min-p 0.5`), I rebuilt the draft head from the same inputs with this packer. Only `experts.bin` changes. Against the current head, 8 interleaved pairs on fresh servers: draft acceptance 0.732 against 0.716, and output speed on a 22,785-token prompt +9.4% (8 of 8 pairs). Short prompts and a 99K-token cold read stayed within noise. That head was built before the commit that adds the previous candidates back to the search, which can only lower each block's error.

Python suite: `tools/test_mtp_pack.py` adds 5 tests. Everything else gives the same results as on v0.1.42, including 8 `setup.py` tests that fail in a plain container on both.

En el sitio

Enlaces a install, modelos, releases.