Pull requests / #1771
#1771 prefill: Volta FP16 tensor-core prompt experts, with an IQ4_NL (type 20) down lane
open · @G0-0000 · 0 comentarios · En GitHub
BenchmarksSetup & installAMD / HIPNVIDIA / CUDAModels & quantsSecurityDocumentation
Descripción
Volta (sm_70) runs the prompt experts on FP16 tensor-core tiles, a group per launch (`STRATA_PF_WMMA`), and this carries an IQ4_NL down lane so UD-IQ4_XS packs can use it. This also carries #1401's three FP16 commits unchanged (b0a127e, 167cc4c, 4bc57c9 — authorship preserved), since they are unmerged. **The three #1401 commits** (NoxestNoz / rewin123): - `prefill`: Volta runs the prompt experts on FP16 tensor-core tiles, a group per launch (`STRATA_PF_WMMA`). - `pf_wmma_parity`: `gemm_iq_f16_grouped` against a double reference over the FP16 path's weights. - `bench/results` + `docs/NVIDIA_V100.md`: V100 prompt experts on FP16 tensor cores (+9.6% / +10.3% prompt speed). **The two commits on top** (this pack's contribution): - `src/kernels/cuda/iq_kernels.cu`: a WK=128 IQ4_NL (type 20) dequant lane in `gemm_iq_f16_grouped` (one thread per row × 64-value half, four `block_iq4_nl` per 128-value window), plus the host dispatch. The down product has K=640, which the generic WK=256 path cannot tile (`640 % 256 != 0`); widening the gate alone therefore fails at launch, so the lane is required. - `src/prefill/prefill.cpp`: two gates widened from `mmq_dt == 42` to `(mmq_dt == 42 || mmq_dt == 20)`. - `src/kernels/pf_wmma_parity.cpp`: parity cases — IQ4_NL 2560×640 down (the new lane), and IQ4_NL / IQ3_S / IQ4_XS 1280×2560 gate/up. **Motive.** UD-IQ4_XS packs (down = IQ4_NL) never take the FP16 prompt path today; their down product falls back to MMQ. **Evidence** (V100-SXM2-32GB, UD-IQ4_XS): `pf_wmma_parity` reports **0 failures**, including the new lane. An env-gated trace confirms the 43 IQ4_NL-down layers take the tiles (43 layers × 4 chunks per round). Phase `gemm down` **−24.9%** (5/5 rounds), but the expert-weight copy wait grows **+17.8%** on our streamed setup, so end-to-end is **~flat** there. Stated honestly: the win is in the phase, not the total, on a streamed configuration. **Caveats.** FP16 activations, not bit-identical to q8_1 — same class as the rest of that path. The sm_60 multi-arch issue discussed in #1401 is not addressed here (we build sm_70).
En el sitio
Enlaces a install, modelos, releases.