#4366 [Engine] DeepSeek prompt: the busy experts take FP16 GEMMs, not the tiles (DS4_FAT_ROWS, opt-in)

closed · @lyesrock · 0 Kommentare · Auf GitHub

AMD / HIPNVIDIA / CUDAModels & quants

Beschreibung

### Problem and result

The chunked prompt decodes an expert's packed weights again for every 64 of its rows. A popular
expert of a long prompt has hundreds of rows, so its weights are decoded and multiplied several
times over. `DS4_FAT_ROWS=N` (default 0: the path is unchanged, the packed tiles run exactly as
before) takes the experts of N rows or more onto one other route: their weights decoded once to the
FP16 values the tiles multiply, the rows gathered, one cuBLAS FP16 GEMM, clamped SwiGLU in FP16.

Measured on the RTX 5090 (sysai2, driver 615.71.10) with the IST-DAS 2.25 bpw pack, this 0.2.7.1
base, ctx 16K, one user, `DS4_FAT_ROWS=64`, ABBA protocol (four fresh boots alternating off/on, a
fresh random 13K-token prompt for every measurement, conversation slots and the page cache cleared
between boots, nice=0, the RAM copy verified at every boot, 11 563 - 11 617 experts):

- first (cold) 13K prompt: patched-off 1 144 / 1 145, on 1 170 / 1 206; and against the
  released 0.1.52 binary itself (a second ABBA, official/on/official/on): official 1 206 /
  1 162, on 1 221 / 1 199. The on side wins every boot: **+2 - 4%**
- further 13K prompts: patched-off 1 561 - 1 594, on 1 642 - 1 689; official 1 579 - 1 623,
  on 1 662 - 1 710 - **+4 - 6%** on both comparisons
- decode: unchanged (short 256: 30 - 32 both; warm ramp 49 - 51 both)

The on side wins all four boots in both regimes. An earlier draft of this PR quoted +39% at 13K
(1 400 to 1 932 - 1 959): that compared a warmed on run against a single cold off run. Corrected
above; sorry for the noise.

The same products, the sums' order differs: a greedy answer can diverge after the first rounding.
12 greedy prompts at temperature 0, boots side by side: off vs the stock 0.1.52 10 of 12 outputs
byte-identical, on vs stock 9 to 10 of 12 - but the stock engine against itself, two boots, gives
9 of 12 on these free-form prompts, so the differences stay inside the stock run-to-run noise (all
differing answers were equivalent wording, same facts, same arithmetic).

### Approach

One expert's weights as FP16 once (`iq::f16_rows`), the rows gathered (`gather_f16`),
`cublasGemmEx`, clamped SwiGLU in FP16 (`swiglu_f16`); `SetArgs skip_ge` leaves those experts out
of the tiles. `skip_ge` sits last in the CUDA/HIP structs, before the Metal-only fields, so the
CUDA/HIP kernel parameter layout keeps upstream's rule (the Metal fields go last). Default 0
changes nothing.

### Validation

Passed: the CUDA build (sm_120) on this base; the ABBA prefill numbers above; the 12-prompt greedy
comparison; decode unchanged on the same boots.

Failed: nothing.

Not run: the HIP and the Metal build of this branch (no ROCm and no Mac on this box) - the touched
files include kernel sources, so a maintainer build there is worth having; a KL/ppl run on this
base (the quality evidence here is the 12-prompt greedy comparison above).

### Limits

The gain is modest (+4 - 6%) at 13K tokens on sm_120: upstream's tiles are themselves much faster
when warm than in the first draft's comparison, so what is left to win is the repeated decode of a
popular expert's weights, not a stalled GEMM. Longer prompts (32K+) and cards with slower
dequant likely keep the larger share of the gain, but that was not measured here.

Mehr auf der Site

Links zu Install, Modellen, Releases.