Pull requests / #600
#600 qsa_prompt_attn: tensor-core kernel for Volta (sm_70, mma.m8n8k4)
closed · @fks · 0 Kommentare · Auf GitHub
BenchmarksNVIDIA / CUDAModels & quantsDocumentation
Beschreibung
On sm_70 `qsa_prompt_attn_batch` refuses the device (#371), so the prompt path runs the QSA attention on the decode kernel, one query at a time with FP32 FMAs: 19.2% of the GPU time of a 28,650-token prompt on a V100 (nsys, 0.1.35, `attn_chunk_kernel<1>`). `prompt_attn_v70_kernel` is the v1 kernel's structure with the two matrix products on `mma.sync.m8n8k4` (FP16 in, FP32 accumulate): q.k with one 64-dim group (= one int8 scale group) per warp and 8 cells per quad-pair, p.v with 16 dims per quad-pair. q and p keep their hi + lo FP16 split, and the four dim groups' partials are added in a fixed order (deterministic). Int8, FP16 and K8V4 KV are handled; Q4_0 KV (mode 4, sm_80+ only) keeps the old kernel. The m8n8k4 fragment layout was measured on a V100 and agrees with turbomind's `SM70_MMA_884`. It is used for compute capability 7.0 - 7.4; `STRATA_PROMPT_ATTN_OLD=1` is the old path. **Accuracy** (`qsa_prompt_attn_parity`, synthetic, V100, 32K context, 2,048 queries per chunk, output scale ~3.6): | KV | error vs FP64, old kernel | error vs FP64, new kernel | time per chunk, old -> new | | --- | ---: | ---: | ---: | | int8 | 2.4e-6 | 5.1e-6 | 38.1 -> 13.4 ms (2.8x) | | fp16 | 1.9e-6 | 5.3e-6 | 37.3 -> 28.3 ms (1.3x) | The 1,500- and 2,100-token contexts also pass. The test now skips its Q4_0 cases below sm_80 instead of exiting 2 there (the dispatcher keeps the old kernel for them). **Prompt speed** (one V100-PCIE-32GB, i9-7960X, CUDA 12.8, `origin/main` at `99f3dbd`, Unsloth `UD-IQ4_XS`, int8 KV; cold server, random-word prompts, greedy; same binary, `STRATA_PROMPT_ATTN_OLD=1` against the default): | Prompt tokens | old path (tok/s) | new kernel (tok/s) | | | ---: | ---: | ---: | ---: | | 7,194 | 1,010 | 1,164 | +15% | | 28,650 | 1,063 | 1,251 | +18% | | 114,338 | 973 | 1,123 | +15% | Decode is unaffected (it does not use this path). **Output checks** - Needle recall (`tools/needle_bench.py`, 8k / 32k / 128k at five depths, a fresh server per configuration): 15 of 15 found with the old path, 15 of 15 with the new kernel. - Greedy output (48 tokens) is identical for the 28,650- and 114,338-token prompts. For the 7,194-token prompt it starts the same and **differs later**: another summation order in a greedy decode. I did not run a teacher-forced top-1 comparison. Not covered: only the Unsloth IQ4_XS pack was run (not the GSQ-RCO quants); one request on one card. Method, profile and raw numbers: `bench/results/2026-10-03-v100-prompt-attn/`; build and kernel overview: `docs/NVIDIA_V100.md` (both in this PR). Independent of the BF16-via-FP16 change in #540: different code files, and it stacks with it (a user measured +9% on top of #540) Written and measured with Claude Sonnet 5.5.
Mehr auf der Site
Links zu Install, Modellen, Releases.