#5551 [Engine] q36 on the qwen35moe 19B-A3B shape: three localized walls with clean data

open · @archadrian · 2 评论 · 去 GitHub 看

BenchmarksNVIDIA / CUDAModels & quantsDocumentation

说明

q36 runs Qwen3.6-35B-A3B well (your docs: unsloth MTP GGUF, 83-117 tok/s decode), so this is not about the engine at its designed shape. We brought up a **qwen35moe 19B-A3B** model (128 experts / top-8, hidden 2048, FF 512, GDN state 128 / conv 4 / 16k+32v heads) and it stops at three specific points. All facts below are from our box (RTX 3090, CUDA 12.8, current main + our qwen3next branch).

**Control first:** the same binary runs Qwen3.6-35B-A3B UD-Q4_K_XL clean on our box at both T=5 and a 457-token multi-chunk prompt (hybrid placement, no NaNs at any of 5,719 traced stages, no faults; 146.6 t/s pp, 22.2 t/s tg on a 3090). So the walls below are specific to this 19B bring-up, not the engine at its designed shape. One caveat that may localize wall 2 further: our 19B file carries BF16 non-expert tensors while the 35B control file is K-quant throughout — the GDN chunk-boundary NaN could be tied to BF16 tensors on the chunked path rather than to the 128-expert shape.

**What loads:** the public GGUFs for this model are Q4_K throughout and are refused at the layout gate (`blk.0.ssm_alpha.weight is Q4_K, expected F32, BF16, F16 or Q8_0`) — expected, but it means no public file loads today. So we built a clean q36 file from the BF16 teacher: TQ expert bytes preserved, non-expert tensors BF16, matrices Q8_0 (tool: `tools/qwen35/q36_from_teacher.py`, in our PR branch). Every tensor verified NaN-free.

**With clean data, three failures, none quant-format related** (they reproduce with TQ experts and with Q6_K experts via `-DSTRATA_Q6K_EXPERTS=ON`):

1. **Small T (5, 17): NaNs inside valid expert rows.** Both the grouped and the MMQ expert paths leave scattered NaNs inside rows that correspond to real routed entries (deterministic columns), plus whole-row NaNs for some entries; the combine then folds them into layer 0. `cap_groups = min(n_ent, n_expert)` for the grouped call looked suspicious to us at small T.
2. **T=502: `l0.gdn_y` goes NaN from row 17 onward** — the first chunk boundary at this GDN shape. Single-chunk GDN (T<=16) is clean.
3. **Then an illegal memory access in the native MMVQ launch.** With `STRATA_Q36_NO_MMQ=1`, compute-sanitizer reports 0 errors, so the fault rides the MMQ path.

Embedding, layer-0 GDN (single chunk), router and the shared expert are clean at T<=16; the MTP block loads; all 128 experts place in VRAM.

Happy to share the clean 19B test files (TQ and Q6_K variants, ~9-13 GB) and a stage-tracer instrumentation (`STRATA_Q36_DEBUG=<file>` prints per-stage NaN counts and positions) if that helps. No action needed from you — filing so the shape gap is on record, and we're glad to help make q35-family shapes a first-class q36 target.

本站相关内容

相关页面的快捷入口。