Pull requests / #2222
#2222 sm_75: fewer launches per decode layer: plan wait folded, fused quantize + interleave (bitwise)
open · @rafatxf · 0 comentarios · En GitHub
Setup & installServer & APINVIDIA / CUDAModels & quants
Descripción
A verify window launches ~35 dependent kernels per layer on its main stream, and on Turing each dependent graph node costs ~0.8 µs of gap (kernel-node nsys). This folds two pairs that always run back to back into one launch each, writing the same bytes: - **The plan's host-flag wait folded into its copy**: 143-amdk's `wait_copy_i32_from_mapped`, gfx12-only until now. On CUDA `STRATA_FUSE_WAIT` selects it: `2` = the plan only (the Turing default), `1` = also the CPU rows' `copy_rows_from_mapped_after`, `0` = neither. The rows' version makes every row block of the copy poll the flag over PCIe and was −1.5 % on an RTX 2080 Ti, so it stays opt-in. `STRATA_FUSE_HITS` is unchanged (off on CUDA; measured neutral here). - **`native_quantize_q8_1_il`**: a projection input's q8_1 blocks and their interleaved copy (`native_q8_1_interleave`) in one kernel. The quantization is `native_quantize_q8_1_kernel`'s; an int of the interleaved copy is four codes gathered from lanes `4p..4p+3` with two shuffles, and the copy's scale is the block's `__low2float(ds)`. Used for the GDN qkv and out-projection inputs and the QSA input and out-projection (`quant_in` in `record_window`); the head keeps its two launches. `STRATA_QIL=0` / `1` forces it off / on; the default is on for sm_75. Both bitwise: the fused quantize was compared byte for byte with quantize + interleave (both buffers) at 2,560 / 6,144 / 10,240 inputs and 2-4 columns, all-zero blocks included. ## Setup RTX 2080 Ti 22 GB (sm_75), i7-6700, CUDA 13.0, SM 1650 MHz, 240 W. Flash-Next, canonical Q2_0 pack, `--max-context 524288`, `--serve`, with the other open Turing decode PRs applied. ## Engine (512 new tokens, two runs each, two separate A/Bs) | A/B 1 | Spanish chat | code | English 32K | |---|---:|---:|---:| | neither | 90.1 t/s | 107.8 t/s | 80.6 t/s | | plan wait folded (`STRATA_FUSE_WAIT=2`) | 90.9 | 108.5 | 81.7 | | A/B 2 | Spanish chat | code | English 32K | |---|---:|---:|---:| | plan wait folded | 88.3\* | 108.0 | 80.8 | | + fused quantize | 90.7 | 109.1 | 80.7 | \* one of the two runs had low draft acceptance (240/337 against ~258/338); the other was 91.4. Small gains (~+1 % and ~+0.5-1 %), but up to three fewer launches per layer in every window, for the same bytes. Part of the Turing set tracked in #2197. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
En el sitio
Enlaces a install, modelos, releases.