Pull requests / #2080
#2080 qsa_select: the FP32 tiled block scores below sm_80 (PR #1091) on by default; STRATA_SELECT_SIMT=0 keeps the warp kernel
open · @imanu86 · 0 comentarios · En GitHub
AMD / HIPNVIDIA / CUDAModels & quantsWindows
Descripción
## Title Issue: none (a default change below sm_80, with the numbers). ## Summary #1091 (konijiwa110) added the FP32 tiled QSA block-scores kernel for the prompt path below sm_80 and it was merged **opt-in** (`STRATA_SELECT_SIMT=1`). This turns it **on by default** on compute capability < 8.0; `STRATA_SELECT_SIMT=0` keeps the warp kernel. sm_80 and newer keep their TF32 tensor-core kernel; HIP is untouched. ## Why On Turing the warp kernel runs at ~1 TFLOPS against ~6.9 for the tiled one, and the QSA block scores are a visible share of a long prompt. #1091 measured +22% at 128K and +44% at 248K prompt speed on an RTX 2080 Ti; here, with the two sm_75 changes of the same series, a 131K prompt on the 2080 Ti stage went from 100 s to 75 s. The ready-made engine is what most Turing users run, and they do not read the environment table. ## What changed `src/kernels/cuda/qsa_select.cu` (`qsa_block_scores_tc`: the `simt` default), `include/strata/kernels/qsa_select.hpp` (the comment), `src/prefill/prefill.cpp` (the log line says which kernel ran). ## Bit for bit No: the tiled kernel sums in another order than the warp kernel, as the opt-in already did. The prompt's selection can differ at the rounding level. ## Measurements RTX 2080 Ti 22 GB + RTX 3060 12 GB, Windows 11, CUDA 12.6, layer split (the 2080 Ti holds layers 16-47), Qwen3.8 IQ3_XXS: 131K prompt 100 s -> 75 s together with the 16-byte dequant and hc16 (separate PRs). TITAN RTX, #1981: the community report ran with the switch on. Built and started on v0.1.43 (RTX 2080 Ti + RTX 3060 layer split, Windows): the engine serves as before, same answer. The prompt-time figures come from this fork's daily build, where the three Turing prefill changes have run together since 0.1.40. Part of a series from a public patch registry (`stack/` in the fork imanu86/Strata-2080Ti), see #2073.
En el sitio
Enlaces a install, modelos, releases.