#3273 [Engine] sycl: opt-in peer prompt row reduction and PUSH return for --expert-parallel-device (stacked on #2205)

说明

## Problem and result

With `--expert-parallel-device` (#1798, prompt path #2205), the peer card returns one FP32 row per routed expert of every prompt token over IPC, and the primary combines them. On two Arc Pro B70s that return dominates the prompt read: at 32K tokens the engine reads 720 tok/s, against 2,078 for a plain two-card layer split of the same model.

Two opt-in switches, both off by default:

- `STRATA_EP_PREFILL_REDUCE=1`: the peer sums its routed experts per token first and returns one FP32 row per token; the primary adds its own routed sum and the shared expert. Changes the FP32 addition order (text can change).
- `STRATA_EP_PREFILL_PUSH=1` (with the reduction): the sum region lives on the primary and the peer writes it through the IPC mapping (posted writes, system-scope fences), instead of the primary reading it from the peer. Transport only: bitwise the reduction's output.

Result on 0.2.0-beta, two B70s, prompt tok/s at 4K / 32K: unset 635 / 720, reduce 993 / 1,264 (+56% / +76%), reduce + push 1,148 / 1,514 (+81% / +110%). Decode unchanged.

## Related discussion

Stacked on #2205 (which builds on #1798), at @recutita's request in recutita/Strata#1. **Only the top three commits are this PR**; the four below them are #1798 and #2205 rebased onto 0.2.0-beta `main` e88ee39d (the only conflict was in `docs/INTEL.md`; `git range-diff` shows #2205's commit unchanged against its current head 4805fa9a). Opened as a draft until #1798 / #2205 have their two-card gate. No maintainer agreement on scope yet.

## Approach

- `sycl/src/prefill/prefill.cpp`: the reduce path when the env is set (rows for the peer's experts summed on the peer with the routing weights, one row per token returned), the push variant allocating the sum region on the primary with the peer as importer. One stderr line when the reduction is on.
- `sycl/src/kernels/cuda/ep_kernels.dp.cpp`, `sycl/include/strata/kernels/ep_kernels.hpp`: the reduce kernel and its posted-write form, a system-scope fence per producer work item before the existing host peer wait.
- `sycl/probe/ep_prefill_reduce.cpp` (new): a two-card probe against an independent FP32 FMA oracle in the original router order.
- `docs/INTEL.md`: one section with the numbers and the probe commands.

Unset: the existing per-expert rows and combine run as before (the env is read once at setup; the unset arm below is this binary with nothing set, not a separate build of the stack below).

## Validation

Setup: 0.2.0-beta `main` e88ee39d + #1798 + #2205 (rebased) + this PR, head 1da1f265 (engine built at 86d8ed20; the top commit is docs only). 2x Intel Arc Pro B70 32 GB, kernel 7.1, Compute Runtime 26.22.38646.4, SYCL AOT `bmg-g31` built with the repository dev image. Qwen3.8-Flash-Next ISTA GSQ-RCO IQ3_XXS, `--expert-parallel-device 1 --prefill 8192 --max-context 65536 --kv int8 --spec 4 --mtp <BF16 draft layer> --mtp-max-t 4 --spec-min-p 0 --stream-experts --expert-cache auto`, Docker `--cap-add SYS_PTRACE`, one slot, greedy, a fresh engine per arm, two fixed 512-token coding replies, then fresh 4K and 32K prompts.

**Passed:**

- `ep-prefill-reduce-probe` and `ep-prefill-reduce-probe --push` (both cards): `PASS cases=26 max_abs=9.53674316e-07` each.
- Engine A/B:

| | 4K read tok/s (ms) | 32K read tok/s (ms) | decode tok/s, two prompts |
|---|---|---|---|
| unset | 635 (5,947) | 720 (42,031) | 80.0 / 81.3 |
| `STRATA_EP_PREFILL_REDUCE=1` | 993 (3,802) | 1,264 (23,934) | 80.6 / 83.8 |
| `+ STRATA_EP_PREFILL_PUSH=1` | 1,148 (3,286) | 1,514 (19,975) | 80.2 / 83.4 |

- Push vs reduce: output SHAs and draft counts identical on both prompts. Reduce vs unset: coherent, text differs (addition order). The engine log shows `EP row reduction enabled` in both reduce arms. Coherence true and per-card health passed after every stop.

**Failed:** none observed.

**Not run:** the `mg_norepeat` / `exact_pp` gate at 4096 and 20000 tokens (offered on #1798; happy to run it on this stack); two slots and `--max-context 200000`; a soak or stability run; CUDA/HIP (no file touched outside `sycl/` and `docs/`).

## Limits

- Even with both switches, expert-parallel prompt reading here (1,514 tok/s at 32K) stays below a plain layer split on 0.2.0 (2,078); expert parallelism's gain is decode (80-84 vs 71-74 tok/s on the same prompts).
- PUSH prints no line of its own; the reduction's line shows either mode.
- Prepared with AI assistance (Claude Code): the code, the measurements and this text were produced by Claude on the submitter's two Arc Pro B70s, at the submitter's request.

🤖 Generated with [Claude Code](https://claude.com/claude-code)