#5595 [Engine] q36: instantiate the grouped down-expert kernel for Q4_K (fmt 12)
closed · @bahooo22 · 0 comentários · No GitHub
BenchmarksServer & APIAMD / HIPNVIDIA / CUDAWindows
Descrição
## Problem and result Models whose down experts are Q4_K (for example Qwen3.6-35B-A3B heretic Q4_K_M: 160 gate/up expert tensors in Q4_K and 80 down tensors split evenly between Q4_K and Q6_K) could not load: `d_qk(12)` returned 0 and `native_expert_supported` refused the file; a type-12 tensor reaching `STRATA_DOWN` would `exit(1)` with a `-DSTRATA_Q6K_EXPERTS=ON` hint that cannot work, because the Q6K branch lacks 12 as well. After this change the same file loads and serves; live numbers below. ## Related discussion none — a missing kernel instantiation, no scope change. ## Approach Add `X(12)` to `STRATA_D_FMTS` in both branches of `src/kernels/cuda/iq_kernels.cu`, and 12 to the (gate/up, down) pair list in `src/kernels/native_grouped_parity.cpp` so the parity gate exercises the new instantiation instead of only compiling it. `Fmt<12>`, `is_iq(12)` and `iq_row_bytes` already existed, so no other site needs a new case. ## Validation **Passed:** - Build: MSVC 14.51 + CUDA 13.3, `-DCMAKE_CUDA_ARCHITECTURES=86`, Windows 11, revision `f9b44b26` (this branch) — compiles clean. - Live serve, single RTX 3060 Ti 8 GiB, `master.txt` prompt, n=256, MTP 0, FA all, super N=4, KV q4_0: 64k 1386.5/16.8; 80k 1273.0/16.5; 96k 1215.6/16.0; 128k 1026.1/14.3; 160k 932.4/13.1; 176k 942.0/14.1; 192k 940.7/13.9 (pp/tg, tok/s). - Needle at 96k: 6 records, 0 errors, 4/6 full-phrase hits, 6/6 by the number. **Failed:** none observed. **Not run:** `native_grouped_parity` for fmt 12 locally (the bench build used `-DSTRATA_BUILD_TESTS=OFF`); HIP, SYCL and Metal builds (no toolchains on this box). The CI parity gate is the first thing to run on this change. ## Limits Only the Q4_K/Q6_K mixed down layout is exercised live; other layouts with Q4_K down experts compile but are not bench-tested here.
No site
Links install, modelos, releases.