Pull requests / #3073
#3073 hip: STRATA_SELECT_SIMT=2 takes the simt2 scorer on HIP too (it silently fell back to the warp kernel)
open · @Vop3p · 0 comments · View on GitHub
BenchmarksMulti-GPUAMD / HIPNVIDIA / CUDAModels & quantsLinux
Description
## Summary `STRATA_SELECT_SIMT=2` (#2071) only reaches the CUDA branch of `qsa_block_scores_tc`. The HIP branch reads only `'1'`, so on AMD cards `=2` silently keeps the warp scorer: no `qsa select: FP32 tiled scorer` line, and no error. This makes `=2` launch `block_scores_simt2_kernel<32, 128>` on HIP too, as the CUDA branch does (the same bits as `=1`). `=1` and unset are unchanged. ## How we found it After updating to 0.1.45 with `STRATA_SELECT_SIMT=2` in our config, 128K prompt reads were 7% lower (1,849 tok/s) and the log had no tiled-scorer line. With `=1`, or with `=2` and this change, they were 1,988-1,994. ## Measured 2x RX 6900 XT (gfx1030), ROCm 10.0, Linux. Flash-Next GSQ-RCO IQ3_S, 262144 context, `--kv int8 --kv-resident 32768`, layer split, `"parallel": 2`. Same binary, `STRATA_SELECT_SIMT=1` vs `=2`, a fresh engine per arm, 10 pairs, order alternating within each pair, one 32K and one 128K prompt per arm: | `STRATA_SELECT_SIMT` | 32K prompt read tok/s, median (range) | 128K prompt read tok/s, median (range) | |---|---:|---:| | `1` | 1,721 (1,719-1,722) | 1,990 (1,987-1,992) | | `2` | 1,722 (1,719-1,723) | 1,995 (1,994-1,998) | Per pair at 128K: +0.2 to +0.4%, median +0.28%, 10 of 10 in the same direction. At 32K the difference is within noise (median +0.09%, 8 of 10). So on gfx1030 the gain over `=1` is small; the point of this PR is that `=2` no longer silently falls back to the warp kernel, which is what cost our 128K reads the 7% above. 0 stalls in the 20 arms. This was measured with our build (0.1.45 with this change and two other local opt-ins, both on in every arm); the binary of this branch alone was used for the parity check below. Confirmation line with `=2`: `qsa select: FP32 tiled scorer (STRATA_SELECT_SIMT=2), tiles simt2 32x128 (#2071)`. ## Tests `tools/batch_test.py` on this branch's binary, `--batch 2 --n 2 --max-new 128 --long-tokens 32768` (the last prompt reads about 32K tokens, i.e. 8,192 indexer blocks, so the top-2,048 selection depends on the scores), `STRATA_IQ_MT_MIN=1`: with `STRATA_SELECT_SIMT=1` and with `=2`, every slot is IDENTICAL to its solo run, and the two dumps (solo and batch tokens of both slots) are identical to each other, as the kernel's comment says (the same bits as `=1`). The `=2` run logs `qsa select: FP32 tiled scorer (STRATA_SELECT_SIMT=2), tiles simt2 32x128 (#2071)`. No other test touches this path on HIP; `tests/hip/native_qsa_score.cpp` covers the native scorer, not `qsa_block_scores_tc`. --- Developed with an AI coding assistant; every number above was measured on 2x RX 6900 XT (gfx1030), ROCm 10.0, Linux.
Related on this site
Quick links to related pages.