#6210 [Engine] Opt-in P2P q8_1 decode push, fused P2P down-projection, and GPU peer sync flag
open · @midhatn · 0 comentários · No GitHub
BenchmarksSetup & installServer & APIMulti-GPUAMD / HIPNVIDIA / CUDAModels & quantsDocumentationWindows
Descrição
Rebased directly onto **`origin/main` (`v0.2.17`, `3142b66b9`)** (independent of `#6023`; commit `145965b72`, preserving `BW-5` `STRATA_L2_PF`). Since `STRATA_P2P_DECODE_SCATTER=1` and `STRATA_P2P_HANDOFF=1` shipped upstream as opt-in switches in `v0.2.17`, this PR cleanly layers the follow-up NVLink P2P decode optimizations as **opt-in switches** behind `STRATA_P2P_DECODE_SCATTER=1`: - **`STRATA_PEER_P2P_Q8=1`** (default off; requires `STRATA_P2P_DECODE_SCATTER=1`): P2P `q8_1` decode activation push from GPU 0 to GPU 1 via per-step peer slots (`d_q8_win_`) + fused P2P down-projection `native_down_lds_p2p_kernel<20/42>` (`native_expert_grouped_p2p`) + zero-DMA $O(n)$ `peer_put_meta` plan upload. - **`STRATA_PEER_GPU_SYNC=1`** (default off; requires `STRATA_P2P_DECODE_SCATTER=1`): Primary-GPU VRAM peer completion flag (`d_peer_flag_` + `wait_flag_ge_abort`), eliminating PCIe host-flag polling inside `PeerExperts`. - Preserves **`STRATA_P2P_HANDOFF=1`** and **`STRATA_ADAPT_OVERLAP`** multi-GPU support from `origin/main` (`3142b66b9`). --- ## Controlled Single-Variable Verification (`v0.2.17`, Locked `200W / 1020 MHz SM / 877 MHz MEM`) > Setup: `2 × Tesla V100-SXM2-16GB` (`NV6` NVLink), `Qwen3.8-Flash-Next` (`512x56B`) `UD-IQ2_XS` + `DFlash2 --spec 6 --draft-conf 0.35`, `--kv int8 --kv-resident 32768 --prefill auto:16384 --peer-device 1`, `64Ki` prompt (`61,694` tokens), `--minimal-env`, **`STRATA_ROUTE_TAIL_SKIP=0`**, `STRATA_PF_WMMA=0`, `STRATA_PF_PEER_STREAM=0`, `STRATA_P2P_DECODE_SCATTER=1`, `STRATA_ADAPT_OVERLAP=1`, `nvidia-smi -pm 1 -pl 200 -lgc 1020,1020 -ac 877,1020` (`sm_min=sm_max=1020 MHz`, `mem_clk=877 MHz`, `ecc=0`). Every run returned `codes=True (3/3)`. ### 1. Interleaved A/B + Opt-In Parity at Default `--peer-reserve-mib 1600` (`max_tokens=256`) | Run (Execution Order) | Binary & Flags (`PEER_P2P_Q8` / `PEER_GPU_SYNC`) | GPU 1 Slots | Prefill (tok/s) | Decode (tok/s) | Drafts Accepted | Output `sha256` prefix | Notes | | --- | --- | ---: | ---: | ---: | ---: | --- | --- | | `s3_A1_base` | `origin/main` (`3142b66b9`) | 8587 | 1,455.5 | 72.9 | 182 / 307 | `affdf99307c6` | Clean baseline (`non_idle_reasons=[]`) | | `s3_B1_pr6210` | `#6210` (`=1 / =1`) | 8584 | 1,458.9 | **77.7** (`+6.6%`) | 184 / 282 | `46990256d0d8` | `d_q8_win_` uses ~5 MiB (-3 slots) | | `s3_A2_base` | `origin/main` (`3142b66b9`) | 8587 | 1,455.8 | 73.0 | 182 / 307 | `affdf99307c6` | Momentary `0x4` power flag (`193.7W`), `sm=1020` held | | `s3_B2_pr6210` | `#6210` (`=1 / =1`) | 8584 | 1,459.4 | **77.8** (`+6.6%`) | 184 / 282 | `46990256d0d8` | `non_idle_reasons=[]` | | `s3_Boff_pr6210` | `#6210` (both **unset**, default off) | 8587 | 1,459.0 | 73.6 | 182 / 307 | `affdf99307c6` | Bit-exact with `origin/main` when unset | | `s3_Bcheckres_pr6210` | `#6210` (`=1 / =1`, `ADAPT_CHECKRES=1`) | 8584 | 1,458.8 | **77.8** | 184 / 282 | `46990256d0d8` | 72 windows, `0` mismatches | ### 2. Equal-Slot Bit-Exact Verification (`8584` Peer Experts Held Constant) Holding GPU 1's peer expert cache strictly equal at **`8584` slots** (`--peer-reserve-mib 1604` on `origin/main` vs `1600` on `#6210`, compensating for `~5 MiB` `d_q8_win_`) isolates `#6210` cleanly (`STRATA_DECODE_TIMING=1`): | Run | GPU 1 Slots | Prefill (tok/s) | Decode (tok/s) | Window Time | `per-layer host` (`plan`) | `adapt wait` | Drafts Accepted | Output `sha256` prefix | | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | --- | | `s3_eq_base_8584` (`origin/main`, `--peer-reserve 1604`) | 8584 | 1,457.4 | 76.7 | 46.36 ms | 11.55 ms (`1.92` plan) | 0.16 ms | 184 / 282 | **`46990256d0d8`** | | `s3_eq_pr6210` (`#6210`, `--peer-reserve 1600`) | 8584 | 1,458.0 | **77.9** (`+1.6%`) | **45.67 ms** (`-1.5%`) | **8.62 ms** (**`-25.4%`**, `1.38` plan `-28.1%`) | **0.01 ms** | 184 / 282 | **`46990256d0d8`** (**bit-exact**) | | `s3_checkres_1024` (`#6210`, `max_tokens=1024`, `CHECKRES=1`) | 8584 | 1,455.7 | **81.0** | — | — | — | 756 / 1122 | `ceb54a947078` (`256 windows, 0 mismatches`) | *(Note: when `--peer-reserve 1605` was tested on `origin/main`, GPU 1 held `8583` slots -> `46.85 ms/window`, `per-layer host 12.00 ms`, `74.9 tok/s`, confirming why `-3` slots at default `1600 MiB` accounts for part of the `+6.6%` default-flag delta and why equal-slot comparison shows a pure bit-exact `-25.4%` host-overhead reduction.)* --- ## Bottlenecks Addressed 1. **`PeerExperts` (`--peer-device N`) host activation bounce, DMA plan upload, separate P2P scatter, and PCIe flag spin:** - **Opt-in P2P `q8_1` activation push (`STRATA_PEER_P2P_Q8=1`):** `doorbell_publish_res_peer` pushes pre-quantized `nat_xq_` directly to per-step peer VRAM slots (`d_q8_win_`) gated by `any_peer`, skipping pinned host FP32 `x` copies whenever every routed expert hits `d_res` or `d_peer_res`. - **Zero-DMA O(n) plan upload (`peer_put_meta` + `d_ident_`):** Replaces $O(n^2)$ `Meta` grouping and `cudaMemcpyAsync` H2D plan transfers with single-pass $O(n)$ grouping, pre-filled peer-device `d_ident_` `[0..CAP)`, and tiered (`512/1024/2048 B`) kernel-arg `peer_put_meta_kernel<128/256/512>` uploads (falling back cleanly to the 4-slot pinned host ring when oversized). - **Fused P2P down-projection + atomic completion (`native_expert_grouped_p2p`):** `native_down_lds_p2p_kernel<20/42>` stages 16-row tiles in shared memory for coalesced `float4` direct NVLink writes into GPU 0 `parts_` (`out[ent_out_row[e] * n_embd + r]`) and raises `*sync_flag = max(*sync_flag, sync_ring)` on the last completing block — eliminating 48 `scatter_rows_f32_flag` launches and HBM2 `d_out_` roundtrips per verify window (`47/48` layers on `UD-IQ2_XS`; falls back cleanly to `native_expert_grouped` + `scatter_rows_f32_flag` on ineligible layers such as layer `4` `Q5_0`). - **Opt-in primary-GPU VRAM completion flag (`d_peer_flag_` + `wait_flag_ge_abort`, `STRATA_PEER_GPU_SYNC=1`):** Moves the P2P completion word into GPU 0 VRAM and spins locally in `wait_flag_ge_abort` while periodically polling mapped `m_peer_flag_` so `Verifier::release_gpu_waits()` unblocks cleanly on shutdown/abort. --- ## Validation ### Passed - **Windows 11 host checks (`145965b72`):** - `python tools/check --only docs,cmake-cpu`: `docs: 6 passed, 0 failed`; `cmake-cpu: 136 passed, 0 failed, 3 skipped`. - `python tools/settings_register.py --check` (exit `0`), `python -m unittest tools.test_settings_register` (`8/8 OK`). - **Dual Tesla V100-SXM2-16GB (Ubuntu, CUDA `12.9`, `sm_70`, locked `200W / 1020 MHz SM / 877 MHz MEM`):** - `cuobjdump --dump-resource-usage`: `native_down_lds_p2p_kernel<42>/<20>` `REG:64 STACK:0 SHARED:256 LOCAL:0`; `native_down_amd_kernel<42,2>` `REG:48 STACK:0 LOCAL:0`. - `STRATA_ADAPT_CHECKRES=1` (`s3_checkres_1024`, `max_tokens=1024`): **`strata serve: res check: 256 windows, 0 with mismatches (0 entries), 0 layer-windows with CPU experts and no activation published`**. - Interleaved A/B + equal-slot `8584`-expert verification (`s3_eq_base_8584` vs `s3_eq_pr6210`): bit-exact `sha=46990256d0d87752`, `codes=True (3/3)`, `ecc=0/0`, `per-layer host 11.55 -> 8.62 ms/window`. ### Failed - None. ### Not run - `256Ki` locked-clock re-run on `v0.2.17` `3142b66b9` (`64Ki` single-variable + `1024`-token `CHECKRES` runs cover decode steady-state + `256`-window parity). ## Limits - `STRATA_PEER_P2P_Q8=1` and `STRATA_PEER_GPU_SYNC=1` are opt-in and require `STRATA_P2P_DECODE_SCATTER=1` + bilateral P2P access; `d_q8_win_` uses ~`5 MiB` of peer VRAM. - `STRATA_PEER_P2P_Q8=1` skips Hadamard-folded gate/up layers (`f.had_gu`), which keep using the existing path. - `native_expert_grouped_p2p` applies to `exp_mode() == 8` layers with `IQ4_NL` (`20`) or `Q2_0` (`42`) down projections (`47/48` layers on `UD-IQ2_XS`), falling back automatically to `native_expert_grouped` + `scatter_rows_f32_flag` on other formats (e.g. layer `4` `Q5_0`).
No site
Links install, modelos, releases.