Pull requests / #1097

#1097 V100: bit-exact GEMV / MoE operator fusions (experimental build only)

open · @ATIVX928 · 0 comentários · No GitHub

BenchmarksSetup & installMulti-GPUNVIDIA / CUDAModels & quants

Descrição

> Resubmitted from #1021: the original was auto-closed by the 2026-10-06 history cleanup. Rebased onto the new `main` (82f46a8); build and parity re-verified on 2x V100.

## What

Operator fusions for the decode GEMV / MoE paths, split out of #627:

- gate/up + SwiGLU in one kernel (`s_gemv_pair`)
- the two-weight BF16 GEMV (`bf16_gemv_pair`, `y1 = w1@x`, `y2 = w2@x`)
- multi-weight GEMVs and an activation-quantize epilogue (`quantize_act_images`)

Every fused output is bitwise what the unfused sequence produced (`memcmp` clean); the fused sequence keeps the unfused kernels' summation order.

## Gating

All fused call sites in `src/core/layer.cpp`, `src/kernels/cuda/shared_expert.cu` and `s2_expert_grouped.cu` are under `#if defined(STRATA_EXPERIMENTAL_SM60)`; the `#else` is main's code verbatim. A non-experimental configure (arch 75, no SM60) builds clean.

## Measured

V100-SXM2-16GB, CUDA 12.8, `fusions_parity --bench` (CUDA events, 200 reps):

| kernel | before | after |
| --- | ---: | ---: |
| expert intermediate (M=1) | 15.6 us | 11.5 us |
| expert intermediate (M=8) | 18.9 us | 11.5 us |
| activation images | 46.1 us | 30.3 us |
| dual GEMV + SwiGLU | 30.3 us | 14.8 us |
| bf16 pair (M=1) | 23.8 us | 13.9 us |
| bf16 pair (M=8) | 23.7 us | 13.9 us |

## 32K end-to-end

Same setup: prefill 1542.8 single / 2202.5 dual vs main's 1557.1 / 2207.0, and decode within the +-15% session noise. The kernel time saved (4-16 us per token) is not visible at this scale, the same conclusion #627 reported. The gain is the kernel time.

## Tests

- `fusions_parity`: 0 failures, all 9 checks bitwise identical.
- The related suites pass (9/9):
  - `s2_gemv_parity`
  - `shared_expert_parity`
  - `s2_gemv_q8_parity`
  - `quantize_act_parity`
  - `s_gemv_parity`
  - `bf16_gemv_parity`
  - `s_gemv_q8k_parity`
  - `s2_expert_grouped_parity`

No site

Links install, modelos, releases.