#5203 [Engine] STRATA_SELECT_F16=1: the prompt path's block scores on FP16 tensor cores (opt-in, sm_80+, 1M read selection 95 -> 62 s)
open · draft · @D-Ogi · 0 评论 · 去 GitHub 看
BenchmarksAMD / HIPNVIDIA / CUDAModels & quantsSecurityWindows
说明
## Problem and result At a 1M-token context, the QSA selection is the largest single phase of a prompt read on an RTX 4090. For a 976K-token prompt it takes 95 of 330 s. About two thirds of that time is the 3xTF32 scorer, `block_scores_tc_kernel`. With `STRATA_SELECT_F16=1`, the same GEMM runs on FP16 tensor cores (m16n8k16) with a three-term hi/lo split. Per 256-query batch at 250,001 blocks: 3.06 -> 1.49 ms. Selection phase: 95 -> 62 s. Prompt read: 2,954 -> 3,270 tok/s. Needles 5/5, decode within the run-to-run spread. The switch is off by default (rounding level). ## Related discussion Scope issue: https://github.com/Niko1221/Strata/issues/5202. #669 (the selection phase at 1M on an RTX 5090). #4118 / #4119 (the two-pass, bitwise approach on the same dispatch). ## Approach `block_scores_f16_kernel<SPLIT>` in `qsa_select.cu`. The keys are the A operand of the MMA. A warp holds 16 blocks in registers for a whole key tile, as hi and lo FP16 parts. The 32 queries x 4 indexer heads of the CTA stay in shared memory in FP32 as the B operand. The kernel builds their hi and lo parts per fragment. SPLIT 3 computes hi.hi + hi.lo + lo.hi. Relu runs per head. The kernel adds the heads in head order in FP32. `qsa_block_scores_tc` dispatches the kernel inside the sm_80+ branch for a reach of 8,192 blocks or more. The write set and the tail block are those of the TF32 kernel. Nothing outlives the launch. The one-term and two-term variants fail the FP64 gate, so they stay bench-only (`STRATA_SELECT_F16_SPLIT=1|2`). CUDA only (`#if !defined(__HIPCC__)`). A startup line names the scorer once. With the variable unset, the dispatch is the previous code. ## Validation Revision: this branch on ffa80d73 (v0.2.10). Windows 11, MSVC 19.44, CUDA 13.0.48, RTX 4090 (sm_89, `CMAKE_CUDA_ARCHITECTURES=89-real`), `STRATA_ENABLE_CUDA=ON STRATA_BUILD_TESTS=ON`. Passed: - `ctest -R "qsa_|batch_rows|decode_cluster"` with the default environment: 15 passed, 0 failed. The same with `STRATA_SELECT_F16=1`: 15 passed, 0 failed. The new ctest `qsa_select_bench_f16` (131K cells, the FP64 gate with the switch on) passes. - `STRATA_SELECT_F16=1 qsa_select_bench 1000000 256 10`: scores 35.27 ms (warp reference) -> 1.49 ms. Gate line: "PASS accuracy vs FP64 (score scale 243): warp kernel max err 2.28e-05, fast scorer max err 0.000282 (1.2e-06 of scale)". Selection line: "selections identical 256/256, cells differing 0.0000%". Without the switch, the same command gives 3.06 ms and a max err of 0.00038 (1.6e-06 of scale). - The same command ran at 40,000, 131,072, 262,144, 524,284, 524,288, 600,000 and 700,000 cells. It also ran for 1, 8, 255, 257, 1,024 and 4,096 queries at 1M cells. Each size ran with the switch off and on. The gate passes everywhere. Max err with the switch on: 1.0e-6 to 1.4e-6 of the scale (TF32: 1.4e-6 to 1.8e-6). The selections match the warp reference as the TF32 kernel's do, with one exception. At 600,000 cells the FP16 kernel differs in 1 of 256 selections (0.0008% of cells), the TF32 kernel in none. At 262,144 cells both differ in 1 of 256. At 1M cells with 1,024 and 4,096 queries both differ in 1 of 1,024 and 4 of 4,096. - `STRATA_SELECT_F16_SPLIT=1` and `=2` (bench only): 1.41 / 1.42 ms, "FAIL accuracy ... 8.4e-05 / 7.6e-05 of scale", selections identical 218/256. This is why the switch runs the three-term kernel. - Engine A/B: Qwen3.8-Flash-Next IQ3_S, `--max-context 1000000 --prefill auto:16384 --ple-io ram`, a 976K-token prompt with five planted codes, `STRATA_PREFILL_TIMING=1`. The same binary ran with the variable off and on. GPU timeline: 330.0 -> 298.0 s. `qsa select`: 95.5 -> 61.8 s. Prompt read: 330.5 -> 298.5 s, 2,954 -> 3,270 tok/s. Every other phase stays within 0.4 s. 5/5 codes found, identical code answers. Decode 65.7 -> 63.9 tok/s on the 300-token summary (the run-to-run spread is about 4 tok/s). The raw lines are in `bench/results/2026-11-04-rtx4090-qsa-select-f16/`. - A local merge of this branch with the one-read top-k change ran the same prompt with both switches on. GPU timeline: 280.9 s. `qsa select`: 46.9 s. Prompt read: 3,470 tok/s. 5/5 codes, identical code answers, decode 61.4 tok/s. Failed: none observed. Not run: - HIP and SYCL builds. No AMD or Intel card here. `#if !defined(__HIPCC__)` excludes the kernel from HIP builds, and the dispatch change sits in the CUDA branch. - Other NVIDIA architectures (sm_80/86/120). - The 10-prompt and 4K-20K identity gate. Reads below 32K cells keep the TF32 kernel, so the gate does not reach the kernel. This PR makes no identity claim. At 1M, the A/B compared the answers only through the planted-code questions. ## Limits One card, one OS, one model. Rounding level by construction: the matrix above shows one selection in 256 that moves at 600,000 cells. The FP64 gate and the selection agreement come from the synthetic keys of `qsa_select_bench` and from one real 976K prompt. Not measured against #4119. The dispatch change would conflict with #4119 textually. The gain is specific to long reads. At a 249K-token read, the selection is about 6% of the read: 3.9 of 64 s. That run used the 0.1.55 tree with this kernel and the one-read top-k switches on. So there is little to gain below ~500K tokens. An LLM (Claude) helped to write the kernel and this text. The author reviewed the kernel and the results by hand. The author asked for more measurements, clarifications and A/B runs during the work. The author built, ran and checked every command above.
本站相关内容
相关页面的快捷入口。