#6419 [Engine] autonomous hardware-synchronized dual-GPU decode window (P2P zero-wait) [Draft speed Lab 230 tok/s+ RFC]

open · draft · @cobra91 · 0 comments · View on GitHub

BenchmarksServer & APIMulti-GPUAMD / HIPNVIDIA / CUDAModels & quantsWindows

Description

> [!NOTE]
> **Draft / Lab RFC**: This pull request is an opt-in experimental banc d'essai shared for community testing and feedback on asymmetric dual-GPU MoE setups. All features are opt-in behind explicit environment flags (`STRATA_ADAPT_OVERLAP`, `STRATA_ADAPT_SIGNAL`, etc.) and default behavior is completely preserved.
> This branch demonstrates the complete end-to-end acceleration combining our earlier isolated PRs (#6089, #6303, #6304, #6382) with the hardware P2P synchronization layer. Once discussed with maintainers, we will decompose and isolate the P2P direct hardware handoff into smaller, focused PRs.
> Live run video & monitor telemetry: https://x.com/cobrax91310/status/2123745844575006791?a=vEQfkD0GhG41No3QjMs&s=20

## Problem and result
On asymmetric dual-GPU systems (such as RTX 5080 16 GB + RTX 4060 Ti 16 GB on a constrained PCIe Gen4 x2 slot), MoE expert offloading suffers from heavy host synchronization latency:
- In Vanilla v0.2.17, the CPU polls and arbitrates every token transfer between devices, creating severe idle gaps and PCIe congestion (up to 46.4% bus saturation), capping sustained 12K decode at 60.6 tok/s.
- With this experimental autonomous P2P hardware-synchronized pipeline, token transfers and completion signals are exchanged directly over PCIe via GPU-mapped flags and fused scatter kernels with zero CPU wait loops.

**Before and after:**
- **Before (Vanilla v0.2.17):** 60.6 tok/s sustained decode on 12K context, 41.0 tok/s cold counting, 63.6 tok/s median counting, 46.4% PCIe traffic.
- **After (Lab P2P Turbo stack):**
  - **Real 12K sustained decode:** 180.7 to 210.6 tok/s (192.1 tok/s average across 70,000 generated tokens, peak at 218.6 tok/s), achieving a **3.5x to 4.0x sustained speedup**.
  - **Counting median:** 236.6 tok/s (+272% vs Vanilla).
  - **Prefill GDDR7 super-chunks:** up to 1,376 tok/s on large prompts (TTFT 2.09s).
  - **PCIe traffic:** down from 46.4% to autonomous P2P direct transfers without host round-trips.

## Related discussion
- Upstream isolated prerequisite PRs:
  - #6089: Cap WDDM expert cache shrink step to 256 MiB
  - #6303: Allow layer-major prefill super-chunks with `--peer-device`
  - #6304: Enable route tail-skip and configurable hot split for `--peer-device`
  - #6382: Windows: preserve page-locked complement with dynamic working-set headroom
  - #6406: Unlock spec 8 window for desktop Blackwell cards
- Issue #4641: Multi-GPU display fix
- Live video & monitor telemetry: https://x.com/cobrax91310/status/2123745844575006791?a=vEQfkD0GhG41No3QjMs&s=20

## Approach
This draft introduces an autonomous, hardware-synchronized P2P execution window for dual-GPU configurations:
1. **Direct GPU-to-GPU PCIe Flag Handoff:** Replaces host-driven CPU polling loops with direct device-side completion signals written across the PCIe bus (`STRATA_ADAPT_SIGNAL=copy`).
2. **Fused Scatter + Signal Kernel:** Implements CUDA kernels (`engine/src/kernels/cuda/verify_kernels.cu`, `elementwise.cu`) that fuse token scatter and signal notification in a single launch, reducing kernel dispatch overhead.
3. **Autonomous Decode Window:** Enables overlap of peer transfer and local expert compute (`STRATA_ADAPT_OVERLAP=1`).
4. **Zero-overhead fallback:** When `STRATA_ADAPT_OVERLAP` is not set, all code paths revert to the standard upstream execution pipeline.
5. **Decomposition plan:** Upon maintainer feedback, we will isolate:
   - Component A: Mapped signal completion flags.
   - Component B: Fused scatter kernels in CUDA backend.
   - Component C: Asymmetric `--peer-hot` defaults and route tail-skipping logic.

### Environment variables used for reproduction
```cmd
:: 1. Dual-GPU P2P Acceleration
set STRATA_ADAPT_OVERLAP=1
set STRATA_ADAPT_SIGNAL=copy
set STRATA_P2P_DECODE_SCATTER=1
set STRATA_ROUTE_TAIL_SKIP=7
set STRATA_GR_DOWN_TG=2

:: 2. Prefill GDDR7 Super-Chunks
set STRATA_PREFILL_LAYER_MAJOR=1
set STRATA_PREFILL_LM_FORCE=1
set STRATA_PREFILL_LM_R=vram
set STRATA_PREFILL_LM_TOKENS=20000

:: 3. Speculation & Memory Headroom
set STRATA_SPEC_GUMBEL=1
set STRATA_DF_PDL=1
set STRATA_PIN_RESERVE_GIB=10
set STRATA_RESIDENT_HEADROOM_GIB=1.5
set STRATA_CKPT_VRAM_SLOTS=0
```

## Validation

**Passed:**
- Windows 11 WDDM (commit 01768f0f2, driver 621.14, Python 3.12):
  - Hardware: RTX 5080 (16 GB, cc 12.0) + RTX 4060 Ti (16 GB, cc 8.9, PCIe Gen4 x2).
  - Model: Swift-Qwen3.8-Flash-Next GSQ-RCO IQ2_XS (65K context).
  - Full load test across 70,000 tokens generated:
    - 5x 12,000-token runs: 180.7, 192.0, 201.1, 210.6, 182.8 tok/s (duration 57-68s each).
    - Median counting: 236.6 tok/s (peak 247.1 tok/s).
    - Code review 8K (temp 0.7, DFlash2 spec 8): 230.4 tok/s.
    - Output verification: 100% greedy identity check against single-GPU reference output.
  - Test suite: `pytest terminal/tests/terminal/` (all 75 tests passed).

**Failed:**
none observed

**Not run:**
Non-CUDA backends (ROCm / Metal / SYCL) on actual hardware execution (build checks run in CI).

## Limits
- Primarily designed for dual-GPU asymmetric topologies where GPU 1 is on a constrained PCIe slot (x2 or x4).
- Direct P2P flags require peer-to-peer PCIe read/write capabilities between devices.

Related on this site

Quick links to related pages.