#6565 [Engine] Add opt-in Hy4-Flash expert parallelism for two DGX Sparks
open · draft · @ddm187 · 0 commentaires · Sur GitHub
BenchmarksServer & APIMulti-GPUAMD / HIPNVIDIA / CUDAModels & quantsLinux
Description
## Problem and result Hy4-Flash can distribute its routed experts across two Linux CUDA hosts. On SQ96 the tested split placed 41.47 GB of expert weights on each Spark. The standalone SQ96 control placed 82.93 GB of expert weights on one device. ## Related discussion Refs #6564. This draft is open for scope discussion. The branch carries an earlier version of the MPI/NCCL transport from #6263. I'd like guidance on the shared transport and branch base. The Hy4 HIP work in #6562 and the Hy4 cache work in #6527 touch some of the same files. ## Approach Build with `-DSTRATA_HY4_DISTRIBUTED=ON` and launch `strata-hy4 --mpi-experts --serve` with one MPI rank per host. The coordinator keeps attention, KV, Engram, vision, and sampling. The worker returns weighted contributions and the coordinator accumulates them in the original router slot order. `HY4_CUDA_GRAPH=1` optionally captures decode and verification compute segments on the coordinator. The DGX Spark guide includes build and launch commands. ## Validation Two DGX Sparks (GB10), Linux ARM64, CUDA 13.0, Open MPI, and NCCL 2.32.3 over RoCE. The branch is based on Strata `v0.2.18` (`6e38f588a`). **Passed** - On `311cdccd8`, the eight MPI and packet tests and the CUDA slot-order regression passed. Matched SQ96 runs produced 20/20 byte-identical FP32 logit rows in an instrumented build and matching greedy outputs in the nine native fixtures with DFly2 off, fixed, and adaptive. - On `3c9bdf028`, the same distributed executable with `HY4_CUDA_GRAPH=0` and `1` matched all nine native and nine fixed DFly2 cases, including prefix reuse and STOP. All 20 paired diagnostic logit rows matched byte for byte. The HTTP text and two Vision fixtures passed. Killing rank 1 during generation ended the MPI job with status 137 and a fresh launch reproduced the candidate's cold output. - A separate CUDA build with `-DSTRATA_HY4_DISTRIBUTED=OFF` built `strata-hy4` and printed `strata-hy4 0.1.64` with `--version`. **Failed during development** - The early prototype used separate partial sums per rank and produced a different greedy token on a short prefix. The tested implementation preserves router slot order. Initial SQ64 benchmark attempts stopped on runner teardown or NCCL log routing. The table uses the complete run after those harness fixes. **Not run** - HIP build and runtime checks for the touched shared Hy4 code, final default-path runtime parity, and long-duration testing. ## Limits Physical coverage is two GB10 hosts. Both arms used SQ64, a 4096-token context, a repeated-token fixture of 3968 input tokens, 64 output tokens, 512-token chunks, adaptive DFly2, `HY4_CUDA_GRAPH=1`, temperature 0, and session reuse off. Each arm had two process starts in A/B/B/A order, with two measured requests per start. Output token IDs matched in all four pairs. | Median | 1 Spark, official Strata 0.2.18 (engine 0.1.64) | 2 Sparks, this branch | | --- | ---: | ---: | | Prefill (tok/s) | 700.7 | 685.9 | | Decode (tok/s) | 65.2 | 66.8 | | Time to first token (s) | 5.67 | 5.79 | <details> <summary>Build and launch commands and individual timings</summary> Set the MPI and NCCL interfaces to the direct link and use the same paths on both hosts. The commands use example hostnames and omit site-specific compiler and library search paths. ```sh cmake -S engine/src/hy4flash/run -B build-hy4 -G Ninja \ -DCMAKE_BUILD_TYPE=Release -DCMAKE_CUDA_ARCHITECTURES=121-real \ -DSTRATA_HY4_DISTRIBUTED=ON cmake --build build-hy4 --parallel ctest --test-dir build-hy4 --output-on-failure cmake --build build-hy4 --target strata-hy4-slot-order-test ./build-hy4/strata-hy4-slot-order-test export HY4_CUDA_GRAPH=1 mpirun --host spark-a:1,spark-b:1 --map-by ppr:1:node --stdin 0 \ -x NCCL_SOCKET_IFNAME -x NCCL_IB_HCA -x HY4_CUDA_GRAPH \ /shared/build-hy4/strata-hy4 --mpi-experts --serve \ --native /shared/models/Hy4-Flash-SQ-64.strata \ --engram /shared/models/Hy4-Flash-Engram-Q8.strata \ --max-context 4096 --chunk 512 --session-reuse 0 --draft dfly2 ``` The single-host control used the same `--serve` arguments with the official engine binary. Each measured request used the native protocol `GEN 64 temperature=0 seed=1 <comma-separated token IDs>`. | Arm and trial | Prefill (ms) | Decode (ms) | Time to first token (ms) | | --- | ---: | ---: | ---: | | One 1 | 5624.2 | 979.0 | 5625.9 | | One 2 | 5636.1 | 976.5 | 5638.2 | | Two 1 | 5793.2 | 915.4 | 5794.8 | | Two 2 | 5787.0 | 1005.3 | 5788.6 | | Two 3 | 5782.5 | 911.8 | 5784.4 | | Two 4 | 5765.5 | 1010.0 | 5767.0 | | One 3 | 5690.6 | 983.7 | 5693.2 | | One 4 | 5726.5 | 1086.5 | 5728.9 | </details>
Sur le site
Liens install, modèles, releases.