Pull requests / #2447

#2447 native_kernels: sm_86 dense Q8_0 GEMVs keep two rows per block (#2441)

closed · draft · @yannickloth · 0 commentaires · Sur GitHub

BenchmarksAMD / HIPNVIDIA / CUDAModels & quantsDocumentationLinux

Description

Fixes the ~20-25% decode regression reported in #2441 on an RTX A3000 Laptop 12 GB (sm_86).

## What this changes

0.1.44 made the dense Q8_0 GEMV (`src/kernels/cuda/native_mmvq.cu`) take one row per block on **every** sm_86 card with `n_out >= 2048`. It was measured on an RTX A4000 (+1-4% at 2-4 columns; 6144 -> 2560 at 6 columns 70% -> 76%). This reverts the sm_86 default to two rows per block. gfx1201 (RDNA4) is unchanged: it keeps one row where it was measured.

`STRATA_NO_MMVQ_ROWS2=1` still forces one row everywhere, sm_86 included, so the rule can be re-measured per card.

## Why

#2441, RTX A3000 Laptop 12 GB (GA104, sm_86, 32 SMs) + i7-12850HX, IQ3_XXS GSQ-RCO pack, `--spec 4`, same request on both engines:

| engine | decode (tok/s) | drafts accepted |
|---|---:|---|
| 0.1.43.1 | 38.1, 38.5, 40.2, 42.1 | 78.9 - 85.7% |
| 0.1.44.1 | 28.7, 32.0, 35.0, 39.7 | 73.9 - 83.6% |
| 0.1.44.1, `STRATA_ROUTE_TAIL_SKIP=0` | 23.7 - 25.8 | 79.1 - 83.1% |

On CUDA/Linux the only decode-path default that changed in 0.1.44 besides the commit batching is this sm_86 rule, and sm_86 is exactly the reporter's card. The dense Q8_0 GDN and attention GEMVs of a verify window are a large share of that window's GPU time, so a per-SKU row choice that is +1-4% on a 48-SM A4000 can be a large loss on a 32-SM A3000.

## Verification

- **Still to run (draft):** the RTX A3000 A/B. Expected: `STRATA_NO_MMVQ_ROWS2=0` (two rows) restores ~0.1.43.1, and `STRATA_NO_MMVQ_ROWS2=1` on 0.1.44.1 reproduces the regression. Please ask the reporter in #2441 to post it before this lands.
- **Answers do not change:** the one-row and two-row kernels keep their own partial sums and reduction, so output is bit-identical.
- **Not built here:** no engine was compiled for this PR; the removed code is the non-HIP branch only, so HIP and SYCL paths are untouched.
- If the A/B instead points at `STRATA_COMMIT_BATCH`, this PR is the wrong fix and will be closed - that is the other default-on 0.1.44 decode change.

Reported and diagnosed on hardware the maintainers do not have (per docs/COMMUNITY_BENCHMARKS.md).

Refs #2441.

Sur le site

Liens install, modèles, releases.