#6210 [Engine] Opt-in P2P q8_1 decode push, fused P2P down-projection, and GPU peer sync flag

open · @midhatn · 0 Kommentare · Auf GitHub

BenchmarksSetup & installServer & APIMulti-GPUAMD / HIPNVIDIA / CUDAModels & quantsDocumentationWindows

Beschreibung

Rebased directly onto **`origin/main` (`v0.2.17`, `3142b66b9`)** (independent of `#6023`; commit `145965b72`, preserving `BW-5` `STRATA_L2_PF`). Since `STRATA_P2P_DECODE_SCATTER=1` and `STRATA_P2P_HANDOFF=1` shipped upstream as opt-in switches in `v0.2.17`, this PR cleanly layers the follow-up NVLink P2P decode optimizations as **opt-in switches** behind `STRATA_P2P_DECODE_SCATTER=1`:

- **`STRATA_PEER_P2P_Q8=1`** (default off; requires `STRATA_P2P_DECODE_SCATTER=1`): P2P `q8_1` decode activation push from GPU 0 to GPU 1 via per-step peer slots (`d_q8_win_`) + fused P2P down-projection `native_down_lds_p2p_kernel<20/42>` (`native_expert_grouped_p2p`) + zero-DMA $O(n)$ `peer_put_meta` plan upload.
- **`STRATA_PEER_GPU_SYNC=1`** (default off; requires `STRATA_P2P_DECODE_SCATTER=1`): Primary-GPU VRAM peer completion flag (`d_peer_flag_` + `wait_flag_ge_abort`), eliminating PCIe host-flag polling inside `PeerExperts`.
- Preserves **`STRATA_P2P_HANDOFF=1`** and **`STRATA_ADAPT_OVERLAP`** multi-GPU support from `origin/main` (`3142b66b9`).

---

## Controlled Single-Variable Verification (`v0.2.17`, Locked `200W / 1020 MHz SM / 877 MHz MEM`)

> Setup: `2 × Tesla V100-SXM2-16GB` (`NV6` NVLink), `Qwen3.8-Flash-Next` (`512x56B`) `UD-IQ2_XS` + `DFlash2 --spec 6 --draft-conf 0.35`, `--kv int8 --kv-resident 32768 --prefill auto:16384 --peer-device 1`, `64Ki` prompt (`61,694` tokens), `--minimal-env`, **`STRATA_ROUTE_TAIL_SKIP=0`**, `STRATA_PF_WMMA=0`, `STRATA_PF_PEER_STREAM=0`, `STRATA_P2P_DECODE_SCATTER=1`, `STRATA_ADAPT_OVERLAP=1`, `nvidia-smi -pm 1 -pl 200 -lgc 1020,1020 -ac 877,1020` (`sm_min=sm_max=1020 MHz`, `mem_clk=877 MHz`, `ecc=0`). Every run returned `codes=True (3/3)`.

### 1. Interleaved A/B + Opt-In Parity at Default `--peer-reserve-mib 1600` (`max_tokens=256`)

| Run (Execution Order) | Binary & Flags (`PEER_P2P_Q8` / `PEER_GPU_SYNC`) | GPU 1 Slots | Prefill (tok/s) | Decode (tok/s) | Drafts Accepted | Output `sha256` prefix | Notes |
| --- | --- | ---: | ---: | ---: | ---: | --- | --- |
| `s3_A1_base` | `origin/main` (`3142b66b9`) | 8587 | 1,455.5 | 72.9 | 182 / 307 | `affdf99307c6` | Clean baseline (`non_idle_reasons=[]`) |
| `s3_B1_pr6210` | `#6210` (`=1 / =1`) | 8584 | 1,458.9 | **77.7** (`+6.6%`) | 184 / 282 | `46990256d0d8` | `d_q8_win_` uses ~5 MiB (-3 slots) |
| `s3_A2_base` | `origin/main` (`3142b66b9`) | 8587 | 1,455.8 | 73.0 | 182 / 307 | `affdf99307c6` | Momentary `0x4` power flag (`193.7W`), `sm=1020` held |
| `s3_B2_pr6210` | `#6210` (`=1 / =1`) | 8584 | 1,459.4 | **77.8** (`+6.6%`) | 184 / 282 | `46990256d0d8` | `non_idle_reasons=[]` |
| `s3_Boff_pr6210` | `#6210` (both **unset**, default off) | 8587 | 1,459.0 | 73.6 | 182 / 307 | `affdf99307c6` | Bit-exact with `origin/main` when unset |
| `s3_Bcheckres_pr6210` | `#6210` (`=1 / =1`, `ADAPT_CHECKRES=1`) | 8584 | 1,458.8 | **77.8** | 184 / 282 | `46990256d0d8` | 72 windows, `0` mismatches |

### 2. Equal-Slot Bit-Exact Verification (`8584` Peer Experts Held Constant)

Holding GPU 1's peer expert cache strictly equal at **`8584` slots** (`--peer-reserve-mib 1604` on `origin/main` vs `1600` on `#6210`, compensating for `~5 MiB` `d_q8_win_`) isolates `#6210` cleanly (`STRATA_DECODE_TIMING=1`):

| Run | GPU 1 Slots | Prefill (tok/s) | Decode (tok/s) | Window Time | `per-layer host` (`plan`) | `adapt wait` | Drafts Accepted | Output `sha256` prefix |
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | --- |
| `s3_eq_base_8584` (`origin/main`, `--peer-reserve 1604`) | 8584 | 1,457.4 | 76.7 | 46.36 ms | 11.55 ms (`1.92` plan) | 0.16 ms | 184 / 282 | **`46990256d0d8`** |
| `s3_eq_pr6210` (`#6210`, `--peer-reserve 1600`) | 8584 | 1,458.0 | **77.9** (`+1.6%`) | **45.67 ms** (`-1.5%`) | **8.62 ms** (**`-25.4%`**, `1.38` plan `-28.1%`) | **0.01 ms** | 184 / 282 | **`46990256d0d8`** (**bit-exact**) |
| `s3_checkres_1024` (`#6210`, `max_tokens=1024`, `CHECKRES=1`) | 8584 | 1,455.7 | **81.0** | — | — | — | 756 / 1122 | `ceb54a947078` (`256 windows, 0 mismatches`) |

*(Note: when `--peer-reserve 1605` was tested on `origin/main`, GPU 1 held `8583` slots -> `46.85 ms/window`, `per-layer host 12.00 ms`, `74.9 tok/s`, confirming why `-3` slots at default `1600 MiB` accounts for part of the `+6.6%` default-flag delta and why equal-slot comparison shows a pure bit-exact `-25.4%` host-overhead reduction.)*

---

## Bottlenecks Addressed

1. **`PeerExperts` (`--peer-device N`) host activation bounce, DMA plan upload, separate P2P scatter, and PCIe flag spin:**
   - **Opt-in P2P `q8_1` activation push (`STRATA_PEER_P2P_Q8=1`):** `doorbell_publish_res_peer` pushes pre-quantized `nat_xq_` directly to per-step peer VRAM slots (`d_q8_win_`) gated by `any_peer`, skipping pinned host FP32 `x` copies whenever every routed expert hits `d_res` or `d_peer_res`.
   - **Zero-DMA O(n) plan upload (`peer_put_meta` + `d_ident_`):** Replaces $O(n^2)$ `Meta` grouping and `cudaMemcpyAsync` H2D plan transfers with single-pass $O(n)$ grouping, pre-filled peer-device `d_ident_` `[0..CAP)`, and tiered (`512/1024/2048 B`) kernel-arg `peer_put_meta_kernel<128/256/512>` uploads (falling back cleanly to the 4-slot pinned host ring when oversized).
   - **Fused P2P down-projection + atomic completion (`native_expert_grouped_p2p`):** `native_down_lds_p2p_kernel<20/42>` stages 16-row tiles in shared memory for coalesced `float4` direct NVLink writes into GPU 0 `parts_` (`out[ent_out_row[e] * n_embd + r]`) and raises `*sync_flag = max(*sync_flag, sync_ring)` on the last completing block — eliminating 48 `scatter_rows_f32_flag` launches and HBM2 `d_out_` roundtrips per verify window (`47/48` layers on `UD-IQ2_XS`; falls back cleanly to `native_expert_grouped` + `scatter_rows_f32_flag` on ineligible layers such as layer `4` `Q5_0`).
   - **Opt-in primary-GPU VRAM completion flag (`d_peer_flag_` + `wait_flag_ge_abort`, `STRATA_PEER_GPU_SYNC=1`):** Moves the P2P completion word into GPU 0 VRAM and spins locally in `wait_flag_ge_abort` while periodically polling mapped `m_peer_flag_` so `Verifier::release_gpu_waits()` unblocks cleanly on shutdown/abort.

---

## Validation

### Passed
- **Windows 11 host checks (`145965b72`):**
  - `python tools/check --only docs,cmake-cpu`: `docs: 6 passed, 0 failed`; `cmake-cpu: 136 passed, 0 failed, 3 skipped`.
  - `python tools/settings_register.py --check` (exit `0`), `python -m unittest tools.test_settings_register` (`8/8 OK`).
- **Dual Tesla V100-SXM2-16GB (Ubuntu, CUDA `12.9`, `sm_70`, locked `200W / 1020 MHz SM / 877 MHz MEM`):**
  - `cuobjdump --dump-resource-usage`: `native_down_lds_p2p_kernel<42>/<20>` `REG:64 STACK:0 SHARED:256 LOCAL:0`; `native_down_amd_kernel<42,2>` `REG:48 STACK:0 LOCAL:0`.
  - `STRATA_ADAPT_CHECKRES=1` (`s3_checkres_1024`, `max_tokens=1024`): **`strata serve: res check: 256 windows, 0 with mismatches (0 entries), 0 layer-windows with CPU experts and no activation published`**.
  - Interleaved A/B + equal-slot `8584`-expert verification (`s3_eq_base_8584` vs `s3_eq_pr6210`): bit-exact `sha=46990256d0d87752`, `codes=True (3/3)`, `ecc=0/0`, `per-layer host 11.55 -> 8.62 ms/window`.

### Failed
- None.

### Not run
- `256Ki` locked-clock re-run on `v0.2.17` `3142b66b9` (`64Ki` single-variable + `1024`-token `CHECKRES` runs cover decode steady-state + `256`-window parity).

## Limits

- `STRATA_PEER_P2P_Q8=1` and `STRATA_PEER_GPU_SYNC=1` are opt-in and require `STRATA_P2P_DECODE_SCATTER=1` + bilateral P2P access; `d_q8_win_` uses ~`5 MiB` of peer VRAM.
- `STRATA_PEER_P2P_Q8=1` skips Hadamard-folded gate/up layers (`f.had_gu`), which keep using the existing path.
- `native_expert_grouped_p2p` applies to `exp_mode() == 8` layers with `IQ4_NL` (`20`) or `Q2_0` (`42`) down projections (`47/48` layers on `UD-IQ2_XS`), falling back automatically to `native_expert_grouped` + `scatter_rows_f32_flag` on other formats (e.g. layer `4` `Q5_0`).

Mehr auf der Site

Links zu Install, Modellen, Releases.