Pull requests / #1771

#1771 prefill: Volta FP16 tensor-core prompt experts, with an IQ4_NL (type 20) down lane

open · @G0-0000 · 0 commentaires · Sur GitHub

BenchmarksSetup & installAMD / HIPNVIDIA / CUDAModels & quantsSecurityDocumentation

Description

Volta (sm_70) runs the prompt experts on FP16 tensor-core tiles, a group per launch (`STRATA_PF_WMMA`), and this carries an IQ4_NL down lane so UD-IQ4_XS packs can use it.

This also carries #1401's three FP16 commits unchanged (b0a127e, 167cc4c, 4bc57c9 — authorship preserved), since they are unmerged.

**The three #1401 commits** (NoxestNoz / rewin123):
- `prefill`: Volta runs the prompt experts on FP16 tensor-core tiles, a group per launch (`STRATA_PF_WMMA`).
- `pf_wmma_parity`: `gemm_iq_f16_grouped` against a double reference over the FP16 path's weights.
- `bench/results` + `docs/NVIDIA_V100.md`: V100 prompt experts on FP16 tensor cores (+9.6% / +10.3% prompt speed).

**The two commits on top** (this pack's contribution):
- `src/kernels/cuda/iq_kernels.cu`: a WK=128 IQ4_NL (type 20) dequant lane in `gemm_iq_f16_grouped` (one thread per row × 64-value half, four `block_iq4_nl` per 128-value window), plus the host dispatch. The down product has K=640, which the generic WK=256 path cannot tile (`640 % 256 != 0`); widening the gate alone therefore fails at launch, so the lane is required.
- `src/prefill/prefill.cpp`: two gates widened from `mmq_dt == 42` to `(mmq_dt == 42 || mmq_dt == 20)`.
- `src/kernels/pf_wmma_parity.cpp`: parity cases — IQ4_NL 2560×640 down (the new lane), and IQ4_NL / IQ3_S / IQ4_XS 1280×2560 gate/up.

**Motive.** UD-IQ4_XS packs (down = IQ4_NL) never take the FP16 prompt path today; their down product falls back to MMQ.

**Evidence** (V100-SXM2-32GB, UD-IQ4_XS): `pf_wmma_parity` reports **0 failures**, including the new lane. An env-gated trace confirms the 43 IQ4_NL-down layers take the tiles (43 layers × 4 chunks per round). Phase `gemm down` **−24.9%** (5/5 rounds), but the expert-weight copy wait grows **+17.8%** on our streamed setup, so end-to-end is **~flat** there. Stated honestly: the win is in the phase, not the total, on a streamed configuration.

**Caveats.** FP16 activations, not bit-identical to q8_1 — same class as the rest of that path. The sm_60 multi-arch issue discussed in #1401 is not addressed here (we build sm_70).

Sur le site

Liens install, modèles, releases.