#6548 [Engine] Add opt-in streamed-KV QSA slot-group batching, pre75 multi-slot attention, and batch SIDE_MIXER

open · draft · @YusenZhang0601 · 0 commentaires · Sur GitHub

BenchmarksSetup & installMulti-GPUAMD / HIPNVIDIA / CUDAModels & quantsWindowsLinux

Description

## Problem and result

In concurrent multi-slot decode with streamed KV (`--batch 4 --kv int8 --kv-resident 32768 --max-context 262144` on Qwen3.8-Flash-Next, 12 QSA layers, 100K context per slot), the QSA selection + KV resolve + attention stage was the single largest non-expert bottleneck on my RTX A6000 (WDDM):

- Even with `#5864` (`STRATA_BATCH_SLOT_RUNS`), each active slot still executes `qsa_block_scores`, `qsa_block_topk`, `qsa_kv_resolve`, and `qsa_decode_attn_batch` **serially slot-by-slot** (4× sequential launches per step across 12 QSA layers, and without `STRATA_BATCH_SLOT_RUNS` it runs row-by-row).
- When `STRATA_ATTN_DAILY=1` (`pre75_attn()`) is enabled, `qsa_decode_attn_slots` and `qsa_decode_attn_batch_rows` decline (`return false` / `per_row = true`), forcing single-query fallback.
- `STRATA_SIDE_MIXER=1/2` is disabled in batch windows (`!batch_rec_`), leaving GDN gate/z projections and QSA indexer/query chains serialized on the main compute stream.

This Draft PR introduces three **opt-in (default OFF)** mechanisms that batch streamed-KV QSA across distinct slots and overlap side-stream mixer work in batch windows:

- **`STRATA_ATTN_PRE75_SLOTS=1`**: adds `attn_chunk_slots_pre75_kernel` and `attn_chunk_rows_pre75_kernel` (bitwise identical to per-query `attn_chunk_kernel_pre75`).
- **`STRATA_BATCH_QSA_STREAM_OPT=1`**: groups contiguous rows of each slot into one cross-slot `qsa_block_scores_slot_groups` launch (reading each slot's `idx_pooled` once per slot), one full-window `qsa_block_topk` launch, one cross-slot `kv_stream_resolve_groups` epoch per distinct streamed slot (with automatic fallback to the existing loop whenever `ntok * max_pages_per_q > n_slots` or `n_slots < 1024`), and batched `qsa_decode_attn_slots`.
- **`STRATA_BATCH_SIDE_MIXER=1`**: unlocks `STRATA_SIDE_MIXER=1/2` in `batch_rec_` windows.

On my `0.1.56` fork (`YusenZhang0601/[email protected]`, `a1c2c904`, RTX A6000 48 GB `sm_86`, Windows WDDM, PCIe Gen3 x16, Dual EPYC 7542, Qwen3.8-Flash-Next IQ3_XXS, 4×100K concurrent decode ABAB, `STRATA_ATTN_DAILY=1`), enabling all three reduced matched-window decode latency from **`39.05 ms` to `33.17 ms` (`-5.88 ms/win`, `-15.1%`)**, marginal slope `(win_ms - 6.64) / avg_rows` from **`5.96` to `4.88 ms/row` (`-18.1%`)**, QSA `sel+resolve+attn` (`STRATA_VERIFY_PROFILE=1`, full batch weighted) from **`9.47 ms` to `2.96 ms` (`-68.7%`)**, and raised system throughput from **`40.2` to `45.0 tok/s` (`+11.9%`)** with **`0 MiB` extra VRAM**.

## Related discussion

none — I am opening this as a **Draft PR** per `CONTRIBUTING.md` to confirm maintainer interest in the scope and knob names before requesting full review or non-CUDA CI validation.

## Approach

All three changes are split into three atomic commits and are **strictly opt-in (default OFF)**, leaving default behavior unchanged when unset:

1. **Commit 1 (`e8cadb76b`) — Pre75 multi-slot / multi-row decode attention (`STRATA_ATTN_PRE75_SLOTS`)**:
   - Factors the body of `attn_chunk_kernel_pre75` into `__device__ attn_chunk_pre75_body` and instantiates `attn_chunk_slots_pre75_kernel` and `attn_chunk_rows_pre75_kernel` (preserving both CUDA `NC=2, DPP=false` and HIP `NC=8, DPP=true` specializations).
   - Adds `static_assert(sizeof(QsaAttnSlotPools) <= 2048)` to stay well within CUDA's 4 KB kernel parameter limit.
   - `qsa_decode_attn_slots` / `qsa_decode_attn_batch_rows` select the pre75 kernels when `STRATA_ATTN_PRE75_SLOTS=1` (or when unset and `STRATA_BATCH_QSA_STREAM_OPT=1`).
2. **Commit 2 (`c12166dfd`) — Slot-grouped QSA scores + grouped streamed-KV resolve (`STRATA_BATCH_QSA_STREAM_OPT`)**:
   - `qsa_select.hpp/.cu`: adds `QsaScoreGroups` (`<= 8` slot groups) and `qsa_block_scores_slot_groups`, where `blockIdx.y` selects the slot group and all contiguous queries of that slot share one read of `idx_pooled` and `idx_dead`.
   - `kv_stream.hpp/.cu`: factors `resolve_kernel` into `__device__ resolve_one_body`, adds `KvResolveGroups` + lean per-kernel parameter structs (`KvResolveMapGroups` 648 B, `KvCopyGroups` 904 B, `static_assert <= 2048`), and launches `resolve_groups_kernel` (`gridDim.x = ng`) + `copy_groups_kernel` (`gridDim.y = ng`) so distinct streamed slots resolve in parallel in one epoch per slot.
   - `verify.cpp`: precomputes `sg`/`rg` before `idx_select` with hard safety guards (rejecting duplicate slots, `ntok > 8`, `sx.n_slots < max(ntok * max_pages_per_q, 1024)`, or `ng > max_groups`, cleanly falling back to the existing `#5864` `slot_runs` / per-row loop).
3. **Commit 3 (`c6bc637d9`) — Batch `SIDE_MIXER` (`STRATA_BATCH_SIDE_MIXER`)**:
   - `verify.cpp`: enables `sidem` and `side_proj` when `batch_rec_ && batch_side_mixer_on()`, coordinating `mjoin_`/`pjoin_` joins across all batch QSA branches (`stream_opt_ok`, `rows_ok`, `g_qsa_slots`, and `#5864` `slot_runs`).

## Validation

**Passed:**
- `origin/main` (`6e38f588a`) + `c6bc637d9` (`strata 0.1.64`, Windows x64, MSVC `14.37.32822`, CUDA `13.0.48`, `sm_86`, CMake + Ninja `-DSTRATA_BUILD_TESTS=ON`):
  - `ninja strata qsa_slots_parity`: **PASS** (`34.9 s`, `strata --version` -> `strata 0.1.64`).
  - `qsa_slots_parity` 3-way control on RTX A6000 (`sm_86`):
    - Negative control (`STRATA_ATTN_DAILY=1`, `STRATA_ATTN_PRE75_SLOTS` unset): `rc=77` (`11.38 s`), 16/16 `SKIP: qsa_decode_attn_slots declined (the pre-sm_75 or gfx906 query-swizzle attention); the engine keeps the per-row calls` (confirms `STRATA_ATTN_DAILY=1` is active and default-OFF preserves existing decline behavior).
    - Positive execution (`STRATA_ATTN_DAILY=1 STRATA_ATTN_PRE75_SLOTS=1`): **`rc=0`** (`10.91 s`), `PASS: 16 cases (4 KV formats x 1/2/4/8 rows of different slots, 0 declined): scores, ids and attention bitwise identical to one call per row`.
    - Baseline control (`STRATA_ATTN_DAILY=0`): **`rc=0`** (`11.59 s`), `PASS: 16 cases (4 KV formats x 1/2/4/8 rows of different slots, 0 declined): scores, ids and attention bitwise identical to one call per row`.
- Historical fork unit & ABAB GPU validation on my `0.1.56` fork (`YusenZhang0601/[email protected]`, `a1c2c904`; RTX A6000 48 GB `sm_86`, Windows 11 WDDM, PCIe Gen3 x16, Dual EPYC 7542, Qwen3.8-Flash-Next IQ3_XXS, `--batch 4 --kv int8 --kv-resident 32768 --max-context 262144 --expert-cache 14300 --pcie-frac 0`, `STRATA_ATTN_DAILY=1`, `0` thermal throttle, `0` CPU interference, `Tmax <= 73°C`):
  - `qsa_slots_parity` (`STRATA_ATTN_DAILY=1 STRATA_ATTN_PRE75_SLOTS=1` and `STRATA_ATTN_DAILY=0`): **16/16 cases bitwise PASS** (4 KV formats `f16/int8/q4/k8v4` × `1/2/4/8` slots, 0 declined).
  - Step-by-step 4×100K ABAB decode results (`0.1.56` fork):

| Arm | Flags | win | avg_rows | win_ms | gpu_wait_ms | slope `(win-6.64)/r` | tok/s | MTP% | vram_free (MiB) |
|---|---|---:|---:|---:|---:|---:|---:|---:|---:|
| `k1_off1` | ALL OFF | 146 | 5.76 | 41.30 | 32.70 | 6.02 | 41.9 | 74.5 | 5733 |
| `k1_ac1` | `+QSA_STREAM_OPT` | 136 | 5.40 | 34.65 | 27.59 | 5.19 | 41.9 | 79.7 | 5791 |
| `k1_acd1` | `+PRE75_SLOTS` | 142 | 5.17 | 32.81 | 25.30 | 5.06 | 43.6 | 79.0 | 5799 |
| `k1_acde1` | `+BATCH_SIDE_MIXER` | 141 | 5.96 | 35.72 | 27.83 | 4.88 | 46.5 | 77.1 | 5819 |
| `k1_off2` | ALL OFF | 135 | 5.44 | 39.05 | 31.35 | 5.96 | 40.2 | 80.1 | 5765 |
| `k1_acde2` | ALL ON | 135 | 5.44 | 33.17 | 25.72 | 4.88 | 45.0 | 80.3 | 5799 |
| `k1_res4096` | ALL ON + `--kv-resident 4096` | 143 | 6.63 | 47.11 | 37.95 | 6.10 | 49.6 | 76.9 | 7451 |

  - Non-MTP 4-lane control (`--batch 4 --spec 2 --lookup-chain 0 --adapt-async 0`, MTP off): slope `5.82/5.84 → 4.86 ms/row` (`-16.6%`).
  - `STRATA_VERIFY_PROFILE=1` (full-batch weighted): `qsa_sel_res_attn` `9.47 → 2.96 ms` (`-68.7%`: `scores+topk 0.61 + kv-resolve 1.50 + attention 0.84`).
  - Functional acceptance suite (262K context, vision, three protocols, tool calls, 4-way concurrent): **10/10 PASS**.
  - Low-residency fallback (`--kv-resident 4096`, `n_slots=1024`): cleanly fell back to per-row resolve without error (`0` overflow, `mem_free=7451 MiB`).

**Failed:**
- none observed

**Not run:**
- End-to-end 4×100K GPU ABAB re-measurement directly on `origin/main` (`engine 0.1.64`, `6e38f588a`); all GPU ABAB numbers above were measured on my `0.1.56` fork (`YusenZhang0601/[email protected]`).
- 24,574-row `kvq` full-corpus sweep (the arithmetic and summation order are unchanged, verified bitwise by `qsa_slots_parity` 16/16).
- HIP, SYCL, Metal, and Linux builds (I only have a Windows RTX A6000 CUDA 13 setup available; HIP specializations in `qsa_decode_attn.cu` and `kv_stream.cu` were written carefully but not compiled on ROCm).

## Limits

- `STRATA_BATCH_QSA_STREAM_OPT=1` requires each streamed slot in a batch window to have `n_slots >= max(ntok * max_pages_per_q, 1024)` (for example `>= 2,056` slots for `ntok = 4`); smaller resident pools such as `--kv-resident 4096` (`n_slots = 1024`) automatically fall back to the existing loop.
- Non-CUDA backends (HIP / Linux) have not been compiled or tested by me; maintainer or community HIP build validation is needed before moving out of Draft.
- All three switches remain opt-in (default OFF) pending cross-architecture validation.

Sur le site

Liens install, modèles, releases.