改动 / #3163
#3163 hip: STRATA_SELECT_SPLIT2=1 - the selection scorer's hi/mid bf16 pieces (3 wmma instead of 6)
open · @vlupilin · 0 评论 · 去 GitHub 看
BenchmarksAMD / HIPNVIDIA / CUDAModels & quants
说明
## Summary Opt-in on top of `STRATA_SELECT_WMMA=1` (AMD gfx11/gfx12 WMMA scorer). The `split3()` already writes each fp32 operand as three bf16 pieces (hi, mid, lo); this keeps the two largest (hi, mid) and drops the three products that carry the lo piece (each <= 2^-16 of the leading term: rel err ~1.5e-5, the class upstream's own WMMA-vs-warp gate accepts at 254/256). Companion to the accepted wide top-k (#1749): together they optimize both stages of the QSA selection pipeline — the top-k that #1749 sped up, and the scorer this PR halves. ## Measurements (gfx1151, Radeon 8060S, Strix Halo) **Kernel level** (`qsa_select_bench`, 256 queries): | blocks | 6-product | split2 | speedup | |---|---:|---:|---:| | 8K | 1.08 ms | 0.53 ms | 2.0x | | 65K | 12.5 ms | 6.6 ms | 1.9x | **Selections identical**: 256/256 at 32K context, 253/256 at 262K (cells 0.0013%; upstream's own WMMA-vs-warp baseline is 254/256). **End-to-end** (98K real prompt, interleaved x2, all arms with `STRATA_GDN_CHUNKED=1`): | arm | tok/s | delta | |---|---:|---:| | mainline | 984.4 | — | | +SPLIT2 | 1002.9 | **+1.9%** | Also measured +1.3% against the four-lanes thread scorer (#1943) on the same card (interleaved T/S/T/S), and split2 covers the sub-16,385-block calls where the thread scorer never engages. ## Scope - One file changed: `src/kernels/cuda/qsa_select.cu` (+35/-14) - `STRATA_SELECT_SPLIT2=1` opt-in; default off = the 6-product kernel's bits unchanged - CUDA does not compile this kernel (it's the HIP WMMA path)
本站相关内容
相关页面的快捷入口。