Pull requests / #2016
#2016 prefill: the QSA top-k radix digit by a cooperative scan (same ids; gfx1151 -5% to -49% at 16K-128K, RTX 3060 -15% to -54% at 4K-64K)
open · @libratechw · 0 commentaires · Sur GitHub
AMD / HIPNVIDIA / CUDAModels & quantsWindows
Description
## Title Issue: none (touches the same file as #1749, a different kernel) ## Summary `block_topk_reg_kernel` picked each radix digit with thread 0 walking up to 255 bins while the other threads waited. Threads 0-255 now prefix-sum the bins in descending order and the one where the count above reaches `width` takes the digit (bin 0 when none of 255..1 does). The digit and the count above are the same, so the selected ids are the same. ## What changed `src/kernels/cuda/qsa_select.cu`, `block_topk_reg_kernel` only. #1749 (open) touches the same file for `block_topk_wide_kernel` (contexts above ~135K cells); the two do not overlap. `qsa_topk_parity <ctx> 2048 10 131072 1` (2,048 queries), ms per call, identical ids in every case (`--selftest` passes on both): | context | gfx1151 before | after | RTX 3060 before | after | |---|---:|---:|---:|---:| | 4K | 0.367 | 0.367 (no change) | 2.12 | 0.975 (-54%) | | 8K | 0.415 | 0.414 (no change) | 2.04 | 1.03 (-49%) | | 16K | 1.30-1.33 | 1.23-1.25 (-5%) | 1.85-2.01 | 1.06-1.14 (-43%) | | 32K | 2.19 | 1.11 (-49%) | 1.94-2.09 | 1.27 (-35%) | | 64K | 2.54 | 1.63 (-36%) | 2.53 | 2.15 (-15%) | | 128K | 4.87 | 3.60-4.00 (-18% to -26%) | 7.29 | 7.29-7.36 (no change) | gfx1151: ROCm 7.14.1, clang 23. RTX 3060: Windows, CUDA 13.0 (MSVC 14.44), sm_86, 2-3 runs each. On gfx1151 4K and 8K do not change (I did not check which path they take); on the 3060 the 128K case does not change (same, not checked). The gain is in the kernel and not yet in the request: UD-Q4_K_XL end to end on gfx1151, 3 interleaved rounds, same text every round, the 128K prompt moved +0.2% (inside the noise). ## Extra Notes No switch: the ids are identical, so it is on for everyone. Not built: SYCL. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
Sur le site
Liens install, modèles, releases.