#6547 [Engine] Add opt-in multi-slot batched MTP drafting (STRATA_BATCH_MTP_FUSED)

open · draft · @YusenZhang0601 · 0 コメント · GitHub で見る

BenchmarksMulti-GPUAMD / HIPNVIDIA / CUDAModels & quantsWindowsLinux

本文

## Problem and result

In `--batch N --batch-mtp` decode mode, `batch_step` in `engine/src/program/generate.cpp` runs `slot_mtp[b]->draft(2, ...)` serially once per active slot after every verify window. With 4 active slots, this reads the entire MTP layer weights (embeddings, `fc_embedding`, `fc_hidden`, Q/K/V/O projections, shared expert, routed MoE experts, and the draft `lm_head`) **4 separate times** per window and performs 4 separate CUDA Graph launches and host stream synchronizations. On my RTX A6000 (48 GB, `sm_86`), this makes `bt_emit` cost ~2.9 ms/window (~8–13% of total decode window time).

This draft PR adds an opt-in multi-slot batched MTP path (`STRATA_BATCH_MTP_FUSED=1`, default OFF) that collapses up to `min(max_t, kFusedGrMaxT, kQsaAttnMaxSlots, 8)` active batch slots into **one** 2-pass batched CUDA Graph on the solo `mtp` drafter (reading all MTP dense projections, shared expert, deduplicated routed experts, and `lm_head` weights **once** instead of `A` times), while updating each slot's own `QsaState` in strict chronological order. It also flushes `BT`/`BDONE` stdout lines before running MTP so streaming clients receive tokens immediately.

On my RTX A6000 48 GB workstation (Windows WDDM, PCIe Gen3 x16, dual EPYC 7542, Qwen3.8-Flash-Next IQ3_XXS, 262K INT8 KV, `--batch 4 --spec 4 --batch-mtp`), measured on my `0.1.56` fork (`YusenZhang0601/[email protected]`, commit `a1c2c904139d5c6a028750141d7a75f35253b76a`, ABAB with 0 thermal throttling and 0 CPU interference):

| 4×100K Metric (row-width matched `avg_rows ≈ 6.0`) | Baseline (`STRATA_BATCH_MTP_FUSED=0`) | `STRATA_BATCH_MTP_FUSED=1` | Delta |
|---|---:|---:|---:|
| `emit_ms` / window | 2.92 ms | **1.85 ms** | **−1.07 ms (−36.6%)** |
| `commit_ms` / window | 1.85 ms | 1.84 ms | −0.01 ms (zero bucket shift) |
| `run_ms` / window | 37.66 ms | 37.69 ms | +0.03 ms (zero `gpu_wait` spillover) |
| Total slope `(win_ms − 6.64) / avg_rows` | 5.952 ms/row | **5.736 ms/row** | **−0.216 ms/row (−3.6%)** |
| MTP acceptance rate | 79.7–80.3% | 78.3–79.3% | within run-to-run variance |
| Extra staging VRAM | 0 B | **0.53 MiB** (`attn_slots=per-row`) | < 1 MiB |

Combined with batched QSA decode (`personal-0.1.56-p2` full-stack ABAB on `0.1.56`, matched 918 tokens / `avg_rows = 5.94–6.00`), `emit_ms` dropped `2.90 -> 1.92 ms` (`-33.8%`) and the functional acceptance suite (262K context, vision, three protocols, tool calls, 4-way concurrent) passed **10/10**.

## Related discussion

none — opening as a Draft PR first to confirm scope, switch naming, and non-CUDA backend expectations with maintainers before requesting review.

## Approach

Changes span 3 files (`engine/include/strata/core/mtp.hpp`, `engine/src/core/mtp.cpp`, `engine/src/program/generate.cpp`):

1. **2-pass zero-gather batching (`record_front_slots` + `record_rest_slots`)**:
   - **Pass 1 (catch-up K/V-only, `N_cu <= A` rows)**: runs only for slots with `keep == 2` (row `0`) and appends K/V into each slot's own `QsaState`.
   - **Pass 2 (accepted-row front + rest, `A` rows)**: runs for all active slots' accepted row `a = keep - 1`, appending K/V chronologically after Pass 1, leaving `R_`, `inj_`, `mixed_`, and `xq_` contiguous for `record_rest_slots` without any gather kernel.
   - `record_rest_slots` executes batched `q_proj`, `q_norm_rope`, per-slot attention, `o_proj`, MLP hyper-connection, `shared_expert_multi` (on `side_` stream concurrently with routed MoE), batched router, `moe_group_resident` + `moe_grouped_s2` across all `T * K` entries (deduplicating experts shared across slots), `native_moe_combine_multi`, `gr_write_multi`, final mixer, batched `lm_head` `native_mmvq`, `argmax_rows`, `row_top_prob_split`, `map_ids`, and `copy_from_mapped_multi` to mapped host output.
2. **Canonical slot ordering for CUDA Graph LRU cache**:
   - `draft_slots()` sorts the active batch items by `slot` pointer before keying the graph cache (LRU cap 64), eliminating 4× redundant graph captures from round-robin `next_slot` rotation (at most `2^4 = 16` graphs in 4-slot steady state).
3. **Lazy staging & VRAM guardrail (`<= 32 MiB`)**:
   - Allocates `0 B` when OFF; when ON, lazily allocates `bm_arena_` (~0.36–0.53 MiB).
   - Multi-slot `qsa_decode_attn_slots` scratch (`bm_attn_scratch_`) is only allocated when `M * attn_scratch_floats_ * 4 <= 32 MiB` (or explicitly forced via `STRATA_BATCH_MTP_ATTN_SLOTS=1`); otherwise it reuses the existing single-row `attn_scratch_` sequentially per slot inside the same CUDA Graph (`0 B` extra attention scratch, avoiding 387–580 MiB allocation if `window == 0` at 262K context).
4. **Strict fallback (`can_draft_slots`)**:
   - Falls back cleanly to the existing per-slot `slot_mtp[b]->draft()` loop whenever `A < 2`, `!stages.empty()` (multi-GPU layer split), Metal, `coupled_active_`, `cell_floor_enabled()`, `STRATA_MTP_CATCHUP_ALL=1`, `STRATA_MTP_TOP2=1`, `rr_res_ != nullptr`, `kv_hybrid`, mismatched `--mtp-q4`/`--mtp-hnorm`/head settings, or near `max_cells`.
   - Preserves per-slot NVMe PLE prefetching overlapped with the GPU graph execution.

Compatibility: Default OFF (`STRATA_BATCH_MTP_FUSED` unset or `0` leaves upstream code path and VRAM usage untouched).

## Validation

**Passed:**
- `origin/main` (`6e38f588a`, engine `0.1.64`, commit `496d73f0a` on `contrib/batched-mtp`) native Windows x86_64 build (MSVC `14.37.32822`, CUDA `13.0.48`, `sm_86`, CMake + Ninja): `ninja strata` PASS (`34.5s`, `strata.exe` `64,926,208 B`, SHA-256 prefix `c56cb901…`), `strata --version` -> `strata 0.1.64` (compile + `--version` smoke; there is no standalone unit-test target for `mtp.cpp`).
- Historical ABAB measurement on my `0.1.56` fork (`YusenZhang0601/[email protected]`, `a1c2c904`, RTX A6000 48 GB `sm_86`, Windows WDDM, PCIe Gen3 x16, 2× EPYC 7542, Qwen3.8-Flash-Next IQ3_XXS 262K INT8 KV, `--batch 4 --spec 4 --batch-mtp`, 0 thermal throttling, 0 CPU interference):
  - Activation log confirmed: `strata mtp: STRATA_BATCH_MTP_FUSED=1 active (chunk_cap=6, staging 0.53 MiB, attn_slots=per-row)`;
  - Standalone ABAB: `emit_ms` `2.92 -> 1.85 ms` (`-1.07 ms/win`, `-36.6%`), zero `commit`/`gpu_wait` regression, functional acceptance suite (262K context, vision, three protocols, tool calls, 4-way concurrent) **10/10 PASS**.

**Failed:**
- none observed

**Not run:**
- End-to-end GPU ABAB re-benchmark on `0.1.64` (`origin/main` `6e38f588a`) — measurements above are from the identical implementation on `0.1.56`.
- 24,574-line `kvq` comparison (MTP only proposes drafts; the target verifier decides every emitted token).
- Linux, HIP, SYCL, and Metal builds (guarded with `#if defined(STRATA_GPU_METAL)` and `#if defined(STRATA_USE_HIP)`; SYCL keeps its own `engine/sycl/src/core/mtp.cpp` and `engine/sycl/src/program/generate.cpp` untouched).

## Limits

- Currently single-GPU only (`stages.empty()`); multi-GPU layer-split falls back to per-slot `draft()`.
- Only handles greedy 1-draft batch MTP rounds (`std::min(max_t - 1, max_drafts) == 1` and `!coupled_active_`); coupled/probabilistic draft sampling and `--batch-df2` keep their existing paths.
- Keep `STRATA_BATCH_MTP_ATTN_SLOTS` unset at long contexts (`window == 0` / 262K) so the automatic `<= 32 MiB` guard stays active.

関連リンク

インストール・モデル・リリースへのサイト内リンク。