#6263 [Engine] Add opt-in MPI/NCCL split inference for two DGX Sparks

open · @ddm187 · 0 comentários · No GitHub

BenchmarksAMD / HIPNVIDIA / CUDAModels & quantsDocumentationLinux

Descrição

## Problem and result

The CUDA split engine expects its GPUs on one host. With `STRATA_SPLIT_DISTRIBUTED=ON`, two DGX Sparks can run one Flash-Next model through the existing engine protocol.

| Workload | One Spark | Two Sparks | Runs per configuration |
| --- | ---: | ---: | ---: |
| Python code, 512 output tokens | 76.93 tok/s | 100.00 tok/s | 4 |
| English LRU-cache explanation, 512 output tokens | 59.33 tok/s | 72.27 tok/s | 4 |
| Cold prefill, 33,280 input tokens | 18.39 s | 14.86 s | 2 |

Values are medians from a 1→2→2→1 run with the same model and DFlash2 settings. Both short prompts produced different continuations across rank counts. All four long-prompt runs retrieved the three requested codes.

## Related discussion

This draft makes the code available for the scope discussion in #6261. Maintainer agreement is pending.

## Approach

Each rank holds part of the attention heads and each expert's feed-forward width. MPI carries engine commands and host results. NCCL carries GPU tensors. Startup requires one process per host. The distributed build also handles Q5_0 routed down tensors in the tested Unsloth UD-IQ4_XS GGUF.

## Validation

Tested `4b4ab879a` on upstream `58ee32277`. The launch and test commands are in `docs/DISTRIBUTED_TP.md`.

**Passed**

- Linux x86-64: the default CUDA split built without MPI/NCCL linkage. `ctest --test-dir build-runtime --output-on-failure` (8 MPI/packet tests) and `PYTHONPATH=terminal python -m unittest discover -s terminal/tests/split -q` (72 tests).
- Two DGX Sparks, Ubuntu 24.04 ARM64, CUDA 13.0, Open MPI 4.1.6 and NCCL 2.32.3. Built with the commands in `docs/DISTRIBUTED_TP.md`. Ran the one- and two-rank NCCL and engine test binaries. The same-host two-rank launch was refused before NCCL initialization.
- Q5_0 matched a pinned ggml decode oracle on 4,194,304 dots and a CPU prefill oracle on two expert tiles. The English DFlash2 loopback API smoke passed 13 checks, including two concurrent requests, disconnect and the next request.

**Failed**

The first decode-oracle compile missed the new include path. Adding `-Iengine/include` fixed the test harness, and the rerun passed.

**Not run**

Byte-for-byte default CUDA split parity against the last release requires two local GPUs on one host. HIP build and physical clusters above two nodes remain untested. I would welcome a same-host two-GPU parity run and a HIP build check.

## Limits

The tested hardware is a two-node Linux CUDA cluster. MTP and image quality were outside this run. Restart the MPI job as a unit after rank loss.

No site

Links install, modelos, releases.