贡献 / #2205
#2205 sycl: with --expert-parallel-device, card N computes its own experts of each prompt chunk
open · @recutita · 0 评论 · 去 GitHub 看
BenchmarksServer & APIMulti-GPUModels & quantsDocumentationWindowsLinux
说明
## Title Issue: none. ## Summary With `--expert-parallel-device N` (#1798), card N now computes its own experts of each prompt chunk. Until now the prompt path ran every expert on card 0 and read card N's experts from the GGUF for every chunk, so a prompt's speed hung on the file cache holding them (a 7200 rpm disk read an 807-token prompt at 12.3 tok/s, reported in #1798). It is on whenever `--expert-parallel-device` is set; `STRATA_EP_PREFILL=0` turns it off and keeps the GGUF reads. Without `--expert-parallel-device` nothing changes. Built on #1798: this branch holds its two commits and this one (c7bb8dd5); only the last is this PR. ### Measured 2x Arc Pro B70 (32 GB each, xe, Linux), Flash-Next IQ3_XXS, Ryzen 9 9950X3D, 60 GB RAM, NVMe, through the server (`sycl/serve/server_intel.py`, 131,072 context, INT8 KV with `--kv-resident 32768`, `--vram-reserve-mib 2048`, `--prefill 4096`), greedy, thinking off. Each value is the median of three fresh documents (256 tokens generated each), two runs per arm, the arms alternated. Hot: one document read first, so the GGUF pages are in the page cache (28-32 GiB of the two files before each document of the old path). Cold: the files' page cache dropped (`posix_fadvise` `DONTNEED`, 0 GiB left) before each document. | prompt | page cache | `STRATA_EP_PREFILL=0` | this PR | change | |---|---|---|---|---| | 627 tokens | hot | 509, 507 tok/s | 761, 761 tok/s | +50% | | 2,441 tokens | hot | 1,091, 1,091 tok/s | 1,218, 1,219 tok/s | +12% | | 627 tokens | cold | 226, 223 tok/s | 757, 762 tok/s | 3.4x | | 2,441 tokens | cold | 590, 584 tok/s | 1,221, 1,225 tok/s | 2.1x | Decode after the prompts (99-107 tok/s) and short chats (75.7-76.2 tok/s) are the same in both arms. Where the time goes (`STRATA_PREFILL_TIMING=1`, one chunk, hot), 2,427 tokens: card 0's expert phases (dequantize, gate/up, down) take 517 ms instead of 1,252 and card N computes its share in 430 ms meanwhile; the combine, which now waits for card N and copies its rows, grows from 56 to 413 ms; the GPU timeline goes from 2,547 to 2,147 ms and the host's GGUF staging from 2,190 ms to none. 612 tokens: card 0's expert phases 924 -> 367 ms, card N 244 ms, combine 15 -> 105 ms, timeline 1,452 -> 955 ms. With the page cache dropped the old path's timeline is 4,311 ms (2,427 tokens) and 2,888 ms (612); this PR's does not change. ## What changed - `sycl/src/prefill/prefill.cpp`, `include/strata/prefill/prefill.hpp`: `Prefill::set_ep_peer`. It reuses the `--peer-device` prompt share (`PeerPrefill`) with card N's `ExpertCache` as the source of its experts. Per MoE layer, card 0 writes the chunk's FP16 activations into a region on card N; card N gathers its rows and runs card 0's FP16 expert path on them (dequantize, oneMKL gate/up, SwiGLU, oneMKL down; this build has no MMQ, which the `--peer-device` share uses), writing the rows into a second region; card 0 waits for card N on the host and copies the rows into its combine input past its caches. Both regions are card N's memory opened on card 0 through Level Zero IPC (`ep_mem.hpp`, as the decode windows' inbox), since dpct gives each card its own context. - `sycl/src/program/generate.cpp`: the call in `--serve`, unless `STRATA_EP_PREFILL=0`. If card N's buffers do not fit (420 MiB of regions at `--prefill 4096`, plus the FP16 path's buffers) the log says so and the GGUF reads stay. - `docs/INTEL.md`: the prompt path line of the expert-parallel section. ## How it was checked - Output: the same four prompts (documents of about 1,000 and 4,000 tokens, two short chats, 256 tokens greedy each) through the server with this PR and with `STRATA_EP_PREFILL=0`, two starts of each: all four outputs byte-identical across the four starts. - The arm with `STRATA_EP_PREFILL=0` logged no expert-parallel prompt line; the other logged `card 1 computes its experts' rows of each prompt chunk (420 MiB of exchange regions)`. - SYCL build (AOT `bmg-g31`), `strata`. ## Extra Notes - `sycl/serve/strata-sycl.sh` passes no variable whose value is `0` to the engine, so `"STRATA_EP_PREFILL": "0"` in a config's `"env"` does not reach it through that launcher (the measurements above used `"00"`, which the engine reads as 0). This is the launcher's rule for every switch and is left alone here. - Card N stays idle for most of a chunk (1,682 of 2,147 ms above) and card 0 waits for it in the combine; overlapping them further is not part of this PR. - Tested only on 2x B70 under xe on Linux with NVMe storage; not on a slower disk, on Arc A-series cards (i915), on Windows, or with cards of two different kinds. Assisted-by: Claude Code with Opus 5.5 medium
本站相关内容
相关页面的快捷入口。