#6581 [Engine] add opt-in zero-spill sm_86 down kernels (STRATA_SM86_DOWN_WS)

open · @nmqx · 0 Kommentare · Auf GitHub

BenchmarksMulti-GPUAMD / HIPNVIDIA / CUDAModels & quantsWindowsLinux

Beschreibung

## Problem and result

On Ampere `sm_86` (RTX 3090, `n_ff = 640`, Flash-Next IQ3_S), the default grouped decode down kernels in `iq_kernels.cu` spill local stack frames on both down formats (`cuobjdump --dump-resource-usage` on `v0.2.19`, CUDA 13.3):

| Down format | Layers (Flash-Next IQ3_S) | Stock `native_down_multi_kernel` | `STRATA_SM86_DOWN_WS=6` |
| --- | --- | --- | --- |
| `IQ4_NL` (type 20) | 39 / 48 | `REG:64 STACK:64 SHARED:2880` | `sm86_down_iq4nl_clean_kernel`: `REG:64 STACK:0 SHARED:2880` |
| `Q2_0` (type 42) | 9 / 48 | `REG:64 STACK:48 SHARED:2880` | `expdec_down_kernel<42, 2>` (`rb=64`): `REG:64 STACK:16 SHARED:2880` |

With opt-in `STRATA_SM86_DOWN_WS=6` (or `=1`; default `0` = stock path unchanged):
- **Kernel micro-bench (`expdec_bench`, RTX 3090 `sm_86`, `T=1..4`):** `memcmp == 0` bit-exact across all 7 Flash-Next quant pairs; `IQ4_NL` down **-1.6% to -7.3%**, `Q2_0` down **-33% to -36%**.
- **End-to-end 2× RTX 3090 decode (`v0.2.19`):** 3-prompt avg `158.6 -> 164.3 tok/s` (**+3.6%**; prose `+14.3`, chat `+10.4`, all-median `+9.7`). On `v0.2.16` with `rt-q4_0` + `STRATA_SM86_DOWN_WS=6`, reached `233.8 / 158.9 / 160.7` (`184.5` avg) and `+9.2 tok/s` `dec@32K`.

## Related discussion

none (small, strictly opt-in `sm_86` kernel switch; default behavior is unchanged).

## Approach

- Single-file change in `engine/src/kernels/cuda/iq_kernels.cu` (+101/-0 lines).
- Adds `sm86_down_iq4nl_clean_kernel` (`__launch_bounds__(256, 4)`, 32 sub-warps × 8 lanes, `rb=32`) reusing existing `ed_stage_entry<ED_HB, false>` and `ed_d20_rows<1, NC>` to stage up to `ED_NC = 4` hidden `q8_1` rows in `2,880 B` shared memory without spilling to local memory.
- Routes `Q2_0` (type 42) to existing `expdec_down_kernel<42, 2>` with `rb=64` (2 rows/lane reuse, `STACK:16` vs `48`).
- Gated behind `STRATA_SM86_DOWN_WS` (`0` default = off; `1` or `6` = both `IQ4_NL` + `Q2_0`; `5` = `Q2_0`-only). CUDA-graph capture compatible (zero host syncs, guarded by `L.n_ff == 32 * ED_HB && L.had_block == 0`).

## Validation

**Passed:**
- **Build & resource check:** Windows 11, CUDA 13.3.73, MSVC 19.50, Ninja, `-DCMAKE_CUDA_ARCHITECTURES=86 -DSTRATA_ENABLE_CUDA=ON -DSTRATA_Q6K_EXPERTS=ON` on `v0.2.19` (`c6f6427d9` + commit). Clean `[171/171]` link; `cuobjdump --dump-resource-usage iq_kernels.cu.obj` confirmed `sm86_down_iq4nl_clean_kernel` = `REG:64 STACK:0 SHARED:2880` vs stock `native_down_multi_kernel<20>` = `REG:64 STACK:64`, and `expdec_down_kernel<42,2>` = `REG:64 STACK:16` vs stock `native_down_multi_kernel<42>` = `REG:64 STACK:48`.
- **Bit-exact parity (`expdec_bench`, RTX 3090):** `memcmp == 0` (0 bit diffs) across all 7 Flash-Next pairs (`22/20`, `18/20`, `21/20`, `23/20`, `22/42`, `18/42`, `21/42`) at `T=1..4`.
- **End-to-end 2× RTX 3090 (Flash-Next GSQ-RCO IQ3_S, `--pipeline-windows 2 --spec 3 --layer-split 24`, temp=0, 3 reps, `v0.2.19`):**

| Run | `STRATA_SM86_DOWN_WS` | Code / Prose / Chat (tok/s) | All (med) | 3-avg |
| --- | --- | --- | ---: | ---: |
| `v0219-q4_0-ws0` | `0` (stock) | `203.2 / 135.6 / 136.9` | `142.5` | `158.6` |
| `v0219-q4_0-ws6` | `6` | `195.7 / 149.9 / 147.3` | **`152.2`** | **`164.3`** (+3.6%) |

**Failed:**
- none observed.

**Not run:**
- HIP (`gfx906`/RDNA), SYCL, Metal, and Linux builds/runs (no AMD/Intel/macOS hardware on this host).
- Non-`sm_86` NVIDIA architectures (`sm_75`/`sm_89`/`sm_120`) and non-`n_ff=640` geometries.

## Limits

- Applies only to `n_ff = 640`, `had_block == 0`, down formats `IQ4_NL` (20) and `Q2_0` (42).
- Strictly opt-in (`STRATA_SM86_DOWN_WS=0` by default); not proposed as an arch default here without broader multi-card validation.

Mehr auf der Site

Links zu Install, Modellen, Releases.