Pull requests / #1100

#1100 s2_qpn8: opt-in QPN8 expert GEMV on Volta m8n8k4 (M >= 4)

open · @ATIVX928 · 0 comments · View on GitHub

BenchmarksSetup & installMulti-GPUAMD / HIPNVIDIA / CUDAModels & quants

Description

> Resubmitted from #1024: the original was auto-closed by the 2026-10-06 history cleanup. Rebased onto the new `main` (82f46a8); build and parity re-verified on 2x V100.

## What

QPN8: a load-time repack of the Q2_0 expert weights at admission plus an m8n8k4 expert GEMV for Volta, split out of #627. The repack doubles the slot at admission and the GEMV builds the per-window A fragment and runs the mma chains; the DP4A path remains the fallback.

The entry point pays off only when a window has M >= 4 expert entries; at M < 4 it is slower, so it stays **opt-in: `STRATA_QPN8=1`, default off**.

## Gating

The m8n8k4 code is compiled under `#if !defined(__HIPCC__) && defined(STRATA_EXPERIMENTAL_SM60)` with stubs that fall back to DP4A elsewhere. The ready-made engine is unchanged.

## Measured

V100-SXM2-16GB, CUDA 12.8, `s2_qpn8_parity --bench`, per window:

| entries | DP4A | QPN8 | ratio |
| --- | ---: | ---: | ---: |
| 10x8 | 109.3 us | 90.6 us | 1.21x |
| 10x4 | 73.5 us | 83.7 us | 0.88x |
| 10x2 | 56.2 us | 77.0 us | 0.73x |
| 10x1 | 50.1 us | 75.6 us | 0.66x |

## Tests

- `s2_qpn8_parity` PASS with `STRATA_QPN8=1`: the quantized intermediates are byte-identical (0 of 30600 bytes differ), the FP32-level outputs are within their bound (worst abs 9.5e-6).
- `s2_expert_grouped_parity` still bitwise PASS.
- Default (QPN8 unset) skips/PASS.

## 32K end-to-end

Same setup with `STRATA_QPN8=1`: prefill 1544.4 single / 2197.0 dual vs main's 1557.1 / 2207.0, and decode within its +-15% session noise. The MTP verify window is around M=5, between the measured 10x4 (0.88x) and 10x8 (1.21x) points. The entry point stays opt-in.

## Known gap

The per-stage sizing loop for a layer split (~`generate.cpp:3514`) still prices `lay.max_blob`, as in the original #627 commit, so a Volta layer split with `STRATA_QPN8=1` can over-pick a stage and fail `open`. The rest of the admission/verify/generate accounting is updated.

Related on strata.com

Editorial links to help you install, pick models, or read release notes — not part of the upstream thread.