Pull requests / #2222

#2222 sm_75: fewer launches per decode layer: plan wait folded, fused quantize + interleave (bitwise)

open · @rafatxf · 0 comentarios · En GitHub

Setup & installServer & APINVIDIA / CUDAModels & quants

Descripción

A verify window launches ~35 dependent kernels per layer on its main stream, and on Turing each dependent graph node costs ~0.8 µs of gap (kernel-node nsys). This folds two pairs that always run back to back into one launch each, writing the same bytes:

- **The plan's host-flag wait folded into its copy**: 143-amdk's `wait_copy_i32_from_mapped`, gfx12-only until now. On CUDA `STRATA_FUSE_WAIT` selects it: `2` = the plan only (the Turing default), `1` = also the CPU rows' `copy_rows_from_mapped_after`, `0` = neither. The rows' version makes every row block of the copy poll the flag over PCIe and was −1.5 % on an RTX 2080 Ti, so it stays opt-in. `STRATA_FUSE_HITS` is unchanged (off on CUDA; measured neutral here).
- **`native_quantize_q8_1_il`**: a projection input's q8_1 blocks and their interleaved copy (`native_q8_1_interleave`) in one kernel. The quantization is `native_quantize_q8_1_kernel`'s; an int of the interleaved copy is four codes gathered from lanes `4p..4p+3` with two shuffles, and the copy's scale is the block's `__low2float(ds)`. Used for the GDN qkv and out-projection inputs and the QSA input and out-projection (`quant_in` in `record_window`); the head keeps its two launches. `STRATA_QIL=0` / `1` forces it off / on; the default is on for sm_75.

Both bitwise: the fused quantize was compared byte for byte with quantize + interleave (both buffers) at 2,560 / 6,144 / 10,240 inputs and 2-4 columns, all-zero blocks included.

## Setup

RTX 2080 Ti 22 GB (sm_75), i7-6700, CUDA 13.0, SM 1650 MHz, 240 W. Flash-Next, canonical Q2_0 pack, `--max-context 524288`, `--serve`, with the other open Turing decode PRs applied.

## Engine (512 new tokens, two runs each, two separate A/Bs)

| A/B 1 | Spanish chat | code | English 32K |
|---|---:|---:|---:|
| neither | 90.1 t/s | 107.8 t/s | 80.6 t/s |
| plan wait folded (`STRATA_FUSE_WAIT=2`) | 90.9 | 108.5 | 81.7 |

| A/B 2 | Spanish chat | code | English 32K |
|---|---:|---:|---:|
| plan wait folded | 88.3\* | 108.0 | 80.8 |
| + fused quantize | 90.7 | 109.1 | 80.7 |

\* one of the two runs had low draft acceptance (240/337 against ~258/338); the other was 91.4. Small gains (~+1 % and ~+0.5-1 %), but up to three fewer launches per layer in every window, for the same bytes.

Part of the Turing set tracked in #2197.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

En el sitio

Enlaces a install, modelos, releases.