Pull requests / #1097
#1097 V100: bit-exact GEMV / MoE operator fusions (experimental build only)
open · @ATIVX928 · 0 comentarios · En GitHub
BenchmarksSetup & installMulti-GPUNVIDIA / CUDAModels & quants
Descripción
> Resubmitted from #1021: the original was auto-closed by the 2026-10-06 history cleanup. Rebased onto the new `main` (82f46a8); build and parity re-verified on 2x V100. ## What Operator fusions for the decode GEMV / MoE paths, split out of #627: - gate/up + SwiGLU in one kernel (`s_gemv_pair`) - the two-weight BF16 GEMV (`bf16_gemv_pair`, `y1 = w1@x`, `y2 = w2@x`) - multi-weight GEMVs and an activation-quantize epilogue (`quantize_act_images`) Every fused output is bitwise what the unfused sequence produced (`memcmp` clean); the fused sequence keeps the unfused kernels' summation order. ## Gating All fused call sites in `src/core/layer.cpp`, `src/kernels/cuda/shared_expert.cu` and `s2_expert_grouped.cu` are under `#if defined(STRATA_EXPERIMENTAL_SM60)`; the `#else` is main's code verbatim. A non-experimental configure (arch 75, no SM60) builds clean. ## Measured V100-SXM2-16GB, CUDA 12.8, `fusions_parity --bench` (CUDA events, 200 reps): | kernel | before | after | | --- | ---: | ---: | | expert intermediate (M=1) | 15.6 us | 11.5 us | | expert intermediate (M=8) | 18.9 us | 11.5 us | | activation images | 46.1 us | 30.3 us | | dual GEMV + SwiGLU | 30.3 us | 14.8 us | | bf16 pair (M=1) | 23.8 us | 13.9 us | | bf16 pair (M=8) | 23.7 us | 13.9 us | ## 32K end-to-end Same setup: prefill 1542.8 single / 2202.5 dual vs main's 1557.1 / 2207.0, and decode within the +-15% session noise. The kernel time saved (4-16 us per token) is not visible at this scale, the same conclusion #627 reported. The gain is the kernel time. ## Tests - `fusions_parity`: 0 failures, all 9 checks bitwise identical. - The related suites pass (9/9): - `s2_gemv_parity` - `shared_expert_parity` - `s2_gemv_q8_parity` - `quantize_act_parity` - `s_gemv_parity` - `bf16_gemv_parity` - `s_gemv_q8k_parity` - `s2_expert_grouped_parity`
En el sitio
Enlaces a install, modelos, releases.