#4366 [Engine] DeepSeek prompt: the busy experts take FP16 GEMMs, not the tiles (DS4_FAT_ROWS, opt-in)
closed · @lyesrock · 0 comentários · No GitHub
AMD / HIPNVIDIA / CUDAModels & quants
Descrição
### Problem and result The chunked prompt decodes an expert's packed weights again for every 64 of its rows. A popular expert of a long prompt has hundreds of rows, so its weights are decoded and multiplied several times over. `DS4_FAT_ROWS=N` (default 0: the path is unchanged, the packed tiles run exactly as before) takes the experts of N rows or more onto one other route: their weights decoded once to the FP16 values the tiles multiply, the rows gathered, one cuBLAS FP16 GEMM, clamped SwiGLU in FP16. Measured on the RTX 5090 (sysai2, driver 615.71.10) with the IST-DAS 2.25 bpw pack, this 0.2.7.1 base, ctx 16K, one user, `DS4_FAT_ROWS=64`, ABBA protocol (four fresh boots alternating off/on, a fresh random 13K-token prompt for every measurement, conversation slots and the page cache cleared between boots, nice=0, the RAM copy verified at every boot, 11 563 - 11 617 experts): - first (cold) 13K prompt: patched-off 1 144 / 1 145, on 1 170 / 1 206; and against the released 0.1.52 binary itself (a second ABBA, official/on/official/on): official 1 206 / 1 162, on 1 221 / 1 199. The on side wins every boot: **+2 - 4%** - further 13K prompts: patched-off 1 561 - 1 594, on 1 642 - 1 689; official 1 579 - 1 623, on 1 662 - 1 710 - **+4 - 6%** on both comparisons - decode: unchanged (short 256: 30 - 32 both; warm ramp 49 - 51 both) The on side wins all four boots in both regimes. An earlier draft of this PR quoted +39% at 13K (1 400 to 1 932 - 1 959): that compared a warmed on run against a single cold off run. Corrected above; sorry for the noise. The same products, the sums' order differs: a greedy answer can diverge after the first rounding. 12 greedy prompts at temperature 0, boots side by side: off vs the stock 0.1.52 10 of 12 outputs byte-identical, on vs stock 9 to 10 of 12 - but the stock engine against itself, two boots, gives 9 of 12 on these free-form prompts, so the differences stay inside the stock run-to-run noise (all differing answers were equivalent wording, same facts, same arithmetic). ### Approach One expert's weights as FP16 once (`iq::f16_rows`), the rows gathered (`gather_f16`), `cublasGemmEx`, clamped SwiGLU in FP16 (`swiglu_f16`); `SetArgs skip_ge` leaves those experts out of the tiles. `skip_ge` sits last in the CUDA/HIP structs, before the Metal-only fields, so the CUDA/HIP kernel parameter layout keeps upstream's rule (the Metal fields go last). Default 0 changes nothing. ### Validation Passed: the CUDA build (sm_120) on this base; the ABBA prefill numbers above; the 12-prompt greedy comparison; decode unchanged on the same boots. Failed: nothing. Not run: the HIP and the Metal build of this branch (no ROCm and no Mac on this box) - the touched files include kernel sources, so a maintainer build there is worth having; a KL/ppl run on this base (the quality evidence here is the 12-prompt greedy comparison above). ### Limits The gain is modest (+4 - 6%) at 13K tokens on sm_120: upstream's tiles are themselves much faster when warm than in the first draft's comparison, so what is left to win is the repeated decode of a popular expert's weights, not a stalled GEMM. Longer prompts (32K+) and cards with slower dequant likely keep the larger share of the gain, but that was not measured here.
No site
Links install, modelos, releases.