#5201 [Engine] One-read top-k: double the sampling stride past 524,288 cells instead of failing every query

closed · @D-Ogi · 0 コメント · GitHub で見る

BenchmarksAMD / HIPNVIDIA / CUDAModels & quantsSecurityWindows

本文

## Problem and result

On a 1M-token prompt, the one-read top-k (`block_topk_sample_kernel`, #2093) fails for every query whose row has more than 131,072 blocks (524,288 cells). Its sample buffer holds 4,096 keys at a stride of 32. When a row needs more samples (`ns > TS_CAP`), the kernel flags the query before it reads anything. The dispatcher then runs `block_topk_wide_kernel<true>`, which reads the row six times. From 524K tokens to the end of the prompt, the switch gives nothing.

With this change, the stride doubles while the samples do not fit: 64 up to 1M cells, 128 up to 2M cells. The same kernel then handles the whole prompt. On an RTX 4090, the top-k of a 256-query batch at 250,001 blocks takes 0.52 ms instead of 1.60 ms. The ids are the ids of the reference.

## Related discussion

#669, observation 2: above ~135K cells the top-k costs more than the block scores, because the kernel reads the row six times. The issue closed on the chunk warning of 0.1.40.2 and left this point open. #2093 added the one-read kernel for Turing. This change only extends its reach. The candidate budget, the threshold and the emit stay the same.

## Approach

`ts` starts at `TS_STRIDE` and doubles until `nb <= TS_CAP * ts` or until `TS_STRIDE_MAX` (128). The sample count of the bound, `ks_want`, follows `ts`. The stride is a function of `nb` alone, so it is the same for the whole CTA.

Off Turing, nothing changes unless the user sets `STRATA_TOPK_ACTIVE_ANY=1 STRATA_TOPK_SAMPLE=1`. On Turing, the kernel is already the default. There, rows past 524,288 cells now take the sampled path instead of the wide fallback. The default path gives the same bits. The parity test below compares the dispatch with and without the bound.

Test: `qsa_topk_active_parity` now sizes its score rows by the context of each case. It adds eight cases. Two sit either side of the first doubling: 524,284 and 524,288 cells. One runs at 600,000 cells. Five run at 1M cells: random scores, all-equal scores, `j % 7` ties, 257 queries and a batch of one. The new ctest `qsa_topk_active_parity_sample` runs every case with `STRATA_TOPK_SAMPLE=1` on any CUDA card.

## Validation

Revision: this branch on ffa80d73 (v0.2.10). Windows 11, MSVC 19.44, CUDA 13.0.48, RTX 4090 (sm_89, `CMAKE_CUDA_ARCHITECTURES=89-real`), `STRATA_ENABLE_CUDA=ON STRATA_BUILD_TESTS=ON`.

Passed:
- `ctest -R "qsa_|batch_rows|decode_cluster"` with the default environment: 15 passed, 0 failed. The same with `STRATA_TOPK_ACTIVE_ANY=1 STRATA_TOPK_SAMPLE=1`: 15 passed, 0 failed. `qsa_topk_active_parity` runs 28 cases. `qsa_topk_active_parity_sample` runs the same 28 cases with the switch on.
- `STRATA_TOPK_ACTIVE_ANY=1 STRATA_TOPK_SAMPLE=1 qsa_select_bench 1000000 256 10`: top-k 6.29 ms (reference) -> 0.52 ms, "register top-k identical to the reference 256/256". Before the change, the same command gave 1.60 ms: the wide fallback, which is also the time of the default dispatch on this build. At 524,288 cells: 2.80 -> 0.32 ms. The ids are also identical at 40,000, 131,072, 262,144, 524,284, 600,000 and 700,000 cells. They are identical for 1, 8, 255, 257, 1,024 and 4,096 queries at 1M cells.
- Engine A/B: Qwen3.8-Flash-Next IQ3_S, `--max-context 1000000 --prefill auto:16384 --ple-io ram`, a 976K-token prompt with five planted codes, `STRATA_PREFILL_TIMING=1`. The same binary ran with the switches off and on. GPU timeline: 329.9 -> 315.7 s. `qsa select`: 95.5 -> 81.2 s. Prompt read: 330.4 -> 316.3 s, 2,955 -> 3,086 tok/s. Every other phase stays within 0.4 s. 5/5 codes found, identical code answers. Decode on the 300-token summary: 64.0 -> 66.8 tok/s. A first run with the switches on gave 55.0 tok/s on that summary. Its prompt phases match the repeat within 0.4 s. In the bench, the one-query top-k is faster with the switches: 0.29 -> 0.14 ms at 1M cells. So the kernel does not explain that run.

Failed: none observed.

Not run:
- HIP and SYCL builds. No AMD or Intel card here. The dispatch change is CUDA-only, and `qsa_select.cu` also compiles for HIP.
- Other NVIDIA architectures.
- The 10-prompt and 4K-20K identity gate. It does not reach this kernel, which starts at 28,672 blocks.

## Limits

Measured on one card and one OS. The sampled bound with a stride of 64 or 128 is sparser. A row with more than 4,096 candidates still uses the wide kernel for that query, as before. So does a row with fewer candidate cells than the width. The parity cases and the 1M prompt showed no such fallback. The fallback is not instrumented.

The default dispatch does not change. On this card, the sampled path also wins at 1M with the switches on (the A/B above). #2093 submitted the kernel for every CUDA card, but measured only an RTX 2080 Ti, so the other cards keep the wide kernel. The prefill bound that the sampled path needs made RTX 5070 64K prompts 1-3% slower (the note in `qsa_block_topk`). A default for sm_89, or a cell threshold for every CUDA card, needs a 64K to 262K A/B on more cards. That is a separate change.

An LLM (Claude) helped to write the change and this text. The author reviewed the diff and the results by hand. The author asked for more measurements, clarifications and A/B runs during the work. The author built, ran and checked every command above.

関連リンク

インストール・モデル・リリースへのサイト内リンク。