#6263 [Engine] Add opt-in MPI/NCCL split inference for two DGX Sparks
open · @ddm187 · 0 Kommentare · Auf GitHub
BenchmarksAMD / HIPNVIDIA / CUDAModels & quantsDocumentationLinux
Beschreibung
## Problem and result The CUDA split engine expects its GPUs on one host. With `STRATA_SPLIT_DISTRIBUTED=ON`, two DGX Sparks can run one Flash-Next model through the existing engine protocol. | Workload | One Spark | Two Sparks | Runs per configuration | | --- | ---: | ---: | ---: | | Python code, 512 output tokens | 76.93 tok/s | 100.00 tok/s | 4 | | English LRU-cache explanation, 512 output tokens | 59.33 tok/s | 72.27 tok/s | 4 | | Cold prefill, 33,280 input tokens | 18.39 s | 14.86 s | 2 | Values are medians from a 1→2→2→1 run with the same model and DFlash2 settings. Both short prompts produced different continuations across rank counts. All four long-prompt runs retrieved the three requested codes. ## Related discussion This draft makes the code available for the scope discussion in #6261. Maintainer agreement is pending. ## Approach Each rank holds part of the attention heads and each expert's feed-forward width. MPI carries engine commands and host results. NCCL carries GPU tensors. Startup requires one process per host. The distributed build also handles Q5_0 routed down tensors in the tested Unsloth UD-IQ4_XS GGUF. ## Validation Tested `4b4ab879a` on upstream `58ee32277`. The launch and test commands are in `docs/DISTRIBUTED_TP.md`. **Passed** - Linux x86-64: the default CUDA split built without MPI/NCCL linkage. `ctest --test-dir build-runtime --output-on-failure` (8 MPI/packet tests) and `PYTHONPATH=terminal python -m unittest discover -s terminal/tests/split -q` (72 tests). - Two DGX Sparks, Ubuntu 24.04 ARM64, CUDA 13.0, Open MPI 4.1.6 and NCCL 2.32.3. Built with the commands in `docs/DISTRIBUTED_TP.md`. Ran the one- and two-rank NCCL and engine test binaries. The same-host two-rank launch was refused before NCCL initialization. - Q5_0 matched a pinned ggml decode oracle on 4,194,304 dots and a CPU prefill oracle on two expert tiles. The English DFlash2 loopback API smoke passed 13 checks, including two concurrent requests, disconnect and the next request. **Failed** The first decode-oracle compile missed the new include path. Adding `-Iengine/include` fixed the test harness, and the rerun passed. **Not run** Byte-for-byte default CUDA split parity against the last release requires two local GPUs on one host. HIP build and physical clusters above two nodes remain untested. I would welcome a same-host two-GPU parity run and a HIP build check. ## Limits The tested hardware is a two-node Linux CUDA cluster. MTP and image quality were outside this run. Restart the MPI job as a unit after rank loss.
Mehr auf der Site
Links zu Install, Modellen, Releases.