Pull requests / #2447
#2447 native_kernels: sm_86 dense Q8_0 GEMVs keep two rows per block (#2441)
closed · draft · @yannickloth · 0 comments · View on GitHub
BenchmarksAMD / HIPNVIDIA / CUDAModels & quantsDocumentationLinux
Description
Fixes the ~20-25% decode regression reported in #2441 on an RTX A3000 Laptop 12 GB (sm_86). ## What this changes 0.1.44 made the dense Q8_0 GEMV (`src/kernels/cuda/native_mmvq.cu`) take one row per block on **every** sm_86 card with `n_out >= 2048`. It was measured on an RTX A4000 (+1-4% at 2-4 columns; 6144 -> 2560 at 6 columns 70% -> 76%). This reverts the sm_86 default to two rows per block. gfx1201 (RDNA4) is unchanged: it keeps one row where it was measured. `STRATA_NO_MMVQ_ROWS2=1` still forces one row everywhere, sm_86 included, so the rule can be re-measured per card. ## Why #2441, RTX A3000 Laptop 12 GB (GA104, sm_86, 32 SMs) + i7-12850HX, IQ3_XXS GSQ-RCO pack, `--spec 4`, same request on both engines: | engine | decode (tok/s) | drafts accepted | |---|---:|---| | 0.1.43.1 | 38.1, 38.5, 40.2, 42.1 | 78.9 - 85.7% | | 0.1.44.1 | 28.7, 32.0, 35.0, 39.7 | 73.9 - 83.6% | | 0.1.44.1, `STRATA_ROUTE_TAIL_SKIP=0` | 23.7 - 25.8 | 79.1 - 83.1% | On CUDA/Linux the only decode-path default that changed in 0.1.44 besides the commit batching is this sm_86 rule, and sm_86 is exactly the reporter's card. The dense Q8_0 GDN and attention GEMVs of a verify window are a large share of that window's GPU time, so a per-SKU row choice that is +1-4% on a 48-SM A4000 can be a large loss on a 32-SM A3000. ## Verification - **Still to run (draft):** the RTX A3000 A/B. Expected: `STRATA_NO_MMVQ_ROWS2=0` (two rows) restores ~0.1.43.1, and `STRATA_NO_MMVQ_ROWS2=1` on 0.1.44.1 reproduces the regression. Please ask the reporter in #2441 to post it before this lands. - **Answers do not change:** the one-row and two-row kernels keep their own partial sums and reduction, so output is bit-identical. - **Not built here:** no engine was compiled for this PR; the removed code is the non-HIP branch only, so HIP and SYCL paths are untouched. - If the A/B instead points at `STRATA_COMMIT_BATCH`, this PR is the wrong fix and will be closed - that is the other default-on 0.1.44 decode change. Reported and diagnosed on hardware the maintainers do not have (per docs/COMMUNITY_BENCHMARKS.md). Refs #2441.
Related on this site
Quick links to related pages.