#5595 [Engine] q36: instantiate the grouped down-expert kernel for Q4_K (fmt 12)

closed · @bahooo22 · 0 commentaires · Sur GitHub

BenchmarksServer & APIAMD / HIPNVIDIA / CUDAWindows

Description

## Problem and result

Models whose down experts are Q4_K (for example Qwen3.6-35B-A3B heretic Q4_K_M: 160 gate/up expert
tensors in Q4_K and 80 down tensors split evenly between Q4_K and Q6_K) could not load: `d_qk(12)`
returned 0 and `native_expert_supported` refused the file; a type-12 tensor reaching `STRATA_DOWN`
would `exit(1)` with a `-DSTRATA_Q6K_EXPERTS=ON` hint that cannot work, because the Q6K branch lacks
12 as well. After this change the same file loads and serves; live numbers below.

## Related discussion

none — a missing kernel instantiation, no scope change.

## Approach

Add `X(12)` to `STRATA_D_FMTS` in both branches of `src/kernels/cuda/iq_kernels.cu`, and 12 to the
(gate/up, down) pair list in `src/kernels/native_grouped_parity.cpp` so the parity gate exercises the
new instantiation instead of only compiling it. `Fmt<12>`, `is_iq(12)` and `iq_row_bytes` already
existed, so no other site needs a new case.

## Validation

**Passed:**
- Build: MSVC 14.51 + CUDA 13.3, `-DCMAKE_CUDA_ARCHITECTURES=86`, Windows 11, revision `f9b44b26`
  (this branch) — compiles clean.
- Live serve, single RTX 3060 Ti 8 GiB, `master.txt` prompt, n=256, MTP 0, FA all, super N=4, KV q4_0:
  64k 1386.5/16.8; 80k 1273.0/16.5; 96k 1215.6/16.0; 128k 1026.1/14.3; 160k 932.4/13.1;
  176k 942.0/14.1; 192k 940.7/13.9 (pp/tg, tok/s).
- Needle at 96k: 6 records, 0 errors, 4/6 full-phrase hits, 6/6 by the number.

**Failed:** none observed.

**Not run:** `native_grouped_parity` for fmt 12 locally (the bench build used
`-DSTRATA_BUILD_TESTS=OFF`); HIP, SYCL and Metal builds (no toolchains on this box). The CI parity
gate is the first thing to run on this change.

## Limits

Only the Q4_K/Q6_K mixed down layout is exercised live; other layouts with Q4_K down experts compile
but are not bench-tested here.

Sur le site

Liens install, modèles, releases.