#6261 [Engine] Run one Flash-Next model across two DGX Sparks

open · @ddm187 · 1 commentaires · Sur GitHub

BenchmarksNVIDIA / CUDALinux

Description

### Area
[Engine]

### What do you want to do?

Can Strata's CUDA split engine run one Flash-Next model across two DGX Sparks? The existing split configuration expects the GPUs to be visible on one host. With two Sparks, I want both machines to hold and compute on their share of the weights while serving the same request.

I have a prototype that runs one MPI process per host. MPI carries commands and host results, NCCL carries GPU tensors, and the server uses the existing engine protocol. I tested the candidate on two GB10s over their direct link. The Unsloth UD-IQ4_XS model I used has Q5_0 routed down weights, which the current split loader rejects. The proposed build handles those weights as well.

With the same Unsloth UD-IQ4_XS model and DFlash2 settings, median decode speed went from 76.93 to 100.00 tok/s for a Python coding prompt and from 59.33 to 72.27 tok/s for an English LRU-cache explanation. Each configuration had four measured 512-token replies per workload. Cold prefill for a 33,280-token English document took 18.39 s on one Spark and 14.86 s on two, with two runs per configuration. All four long-prompt runs retrieved the three requested codes. Both short prompts produced different continuations across rank counts.

Would an opt-in Linux CUDA build with that scope fit? Draft PR #6263 contains the implementation and validation details.

### What do you do today instead?

I run the Spark build on one host or use a local build of the cluster prototype. The broader multi-node question came up in #1750.

Sur le site

Liens install, modèles, releases.