#3648 [Engine] --peer-device host route: STRATA_PF_PEER_PIECES=1 (opt-in) moves the activations and the sums in 8 pieces (+7 % prompt on 2x V100)

说明

## Problem and result

With `--peer-device` and no P2P between the two cards, every MoE layer of a prompt chunk makes two transfers through mapped host memory. This is the host route: `STRATA_PEER_NO_P2P=1`, or a pair whose P2P test fails (#2666). The primary waits through both transfers:

1. **Activations:** the FP16 activations go down to host memory only after the routing and the host grouping. The peer then reads all of them before its first expert starts.
2. **Sums:** the peer's FP16 per-token sums come back in one piece. The primary's combine reads them from host memory once the whole block has landed.

With `STRATA_PF_PEER_PIECES=1` (opt-in), both transfers go in 8 pieces. The activations start down before the router, and the peer converts each piece as it lands, while the router, the shared expert and the host grouping still run. The sums come back piece by piece, and the primary copies each piece into its own memory as it lands.

On 2x Tesla V100-PCIE-32GB without P2P (Qwen3.8-Flash-Next IQ3_S, 16,384-token chunks):

| prompt | without the variable | `STRATA_PF_PEER_PIECES=1` | per round |
|---|---:|---:|---|
| ~10K tokens | 1,599 tok/s | 1,711 tok/s (**+7.0 %**) | +6.0 to +7.3 %, faster in 4 of 4 |
| ~38K tokens | 1,669 tok/s | 1,793 tok/s (**+7.5 %**) | +6.3 to +7.8 %, faster in 4 of 4 |

The values are medians of 4 rounds. The combine phase, which includes the wait for the peer, went from 3,131 to 1,196 ms on a 37,808-token prompt. The output is bit for bit the same as `main`'s. Without the variable, and on the P2P route, nothing changes.

## Related discussion

none. The host route with per-token sums and FP16 transfers came with #1251. This adds an opt-in way to overlap its two transfers and changes no default.

## Approach

- **Activations.** In an MMQ layer, before the router, the primary records an event on its compute stream. The host route's copy stream (`x_stream`) then copies `mixed_h` to `host_x16` in 8 pieces, with an event after each piece. On the peer, `f16_to_f32_wide` converts each piece into `P.mixed` once its event has fired. These are the same values the existing copy after the grouping produces, so that copy and conversion are skipped for this layer.
- **Ordering of the early send.** It is sent before anyone knows whether this layer has peer rows. `ev_done` is therefore recorded on the peer stream after the last conversion. If the layer turns out to have no peer rows, the primary waits on that event before the next layer can overwrite `host_x16` or `mixed_h`. Otherwise the existing waits cover it.
- **Sums.** The peer converts and copies down each piece separately (`sums_to_f16` per piece, then the D2H copy). A stream on the primary (`s_h`) reads each piece with `f16_to_f32_wide` into `bo`. `bo` is free until the combine writes it, and the stream's first wait already orders it after the previous layer's `gr_write`. The combine waits for the last piece (`ev_h`) and reads `bo` through `moe_combine_peer`. The conversions are exact (FP16 to FP32), and the additions keep their order, so the result is bit for bit the same as `moe_combine_peer16` reading `host_sum16`.
- **The combine kernel.** `moe_combine_peer` is now called with `peer == bo`, so I removed `__restrict__` from those two parameters. Each thread reads its element before it writes it. The kernel's other caller (FP32 sums) gives the same results.
- **Mapped reads.** Both piece reads use kernels on mapped memory, as the host route already does. A copy-engine upload would queue behind the expert stream.
- **Cost.** Without the variable, nothing is created or run. With it, the primary gets one stream and 9 events, and the peer gets 8 events. No new buffers.
- **Piece bounds.** Both sides split the pieces with the existing `handoff_piece()`.

## Validation

All runs were on 2x Tesla V100-PCIE-32GB (PCIe 3.0 x16 each, no NVLink, `SYS` in `nvidia-smi topo`), an Intel Core i9-7960X with 125 GiB RAM, Ubuntu (Linux 6.8.0-51), driver 550.120 and CUDA 12.8. The model was Qwen3.8-Flash-Next GSQ-RCO IQ3_S with a native pack, run with `--prefill auto:16384 --kv int8 --ple-io ram --peer-device 1`. The full command, the environment, the raw runs and the logs are in `bench/results/2026-10-21-peer-pieces/`.

**Passed:**

- CUDA build (sm_70, Release, flags in the bench README) of this branch and of `main` `011c9e60`.
- Prompt throughput: the same binary without and with the variable, each a separate server start, four rounds in alternating order. Each round sent a ~10K-token and a ~38K-token prompt made from the repository's docs. All 16 runs are in `runs.jsonl`, the medians are in the table above, and the variable was faster in every round. Earlier the same day, unchanged `main` measured 1,596 / 1,672 tok/s on the same prompts, the same as this branch without the variable.
- `STRATA_PREFILL_TIMING=1`, one server start per arm:
  - combine 806 → 283 ms (9,884 tokens) and 3,131 → 1,196 ms (37,808 tokens);
  - GPU timeline 5,666 → 5,282 ms and 21,949 → 20,548 ms.
- Output, teacher-forced with `STRATA_LOGPOS`: five prompts of ~34K/24K/16K/8K/3K tokens, top-20 log probabilities at 2,493 positions.
  - With `STRATA_PEER_NO_P2P=1`, this branch is bit for bit identical to `main` at every position, both without and with `STRATA_PF_PEER_PIECES=1` (`logpos-vs-main-noP2P.txt`).
  - With P2P, where the variable has no effect, this branch with the variable set is bit for bit identical to `main` (`logpos-vs-main-P2P.txt`).
- A 17,659-token prompt and a short question with the variable set gave a normal summary and answer (`engine-logs/fin_on-0-smoke.txt`).
- `ctest -j 8` in the CUDA build: 62 passed, 3 skipped, 4 failed (next list). Run alone afterwards, `vmm_test` and `kv_window_parity` passed.

**Failed:**

- `ctest -j 8`: `pf_wmma_parity` and `ple_parity` fail on this machine on unchanged `main` `011c9e60` too (checked the same day), so they are not from this change; I did not investigate them. `kv_window_parity` failed under `-j 8` on both this branch and `main`, and passed when run alone on both. `vmm_test` failed once under `-j 8` on this branch (it passed in `main`'s `-j 8` run), and passed when run alone on this branch and on `main`. Both look like contention between tests sharing the GPUs; I did not investigate further.

**Not run:**

- HIP and SYCL builds. `src/prefill/prefill.cpp` and `src/prefill/kernels.cu` are part of them, so the change is unverified there, and this PR stays a draft until they are built.
- GPUs other than V100, and faster links (PCIe 4.0/5.0). The gain depends on how the link speed compares to the compute speed.
- Decode. The change is in the prompt path only.
- Prompts below ~7K tokens, and chunk sizes other than 16,384 tokens.
- The two halves separately in this final form. On an earlier revision with one variable for each half, at ~38K tokens:
  - the early activations alone gave +6.0 %;
  - the sums in pieces alone gave +1.6 %, and +1.0 % on top of the early activations.

## Limits

- Host route with FP16 transfers only (`STRATA_PF_PEER_F16` on, the default there). The FP32 transfers and the P2P route are unchanged.
- In a layer without peer rows, the activations cross for nothing, and the primary waits for the peer to finish reading them.
- The piece count is fixed at 8. I did not sweep it.
- It is opt-in. I would not propose making it the default without measurements on other no-P2P pairs, for example GeForce cards.

🤖 Generated with [Claude Code](https://claude.com/claude-code)