#6261 [Engine] Run one Flash-Next model across two DGX Sparks
open · @ddm187 · 1 comments · View on GitHub
Description
### Area [Engine] ### What do you want to do? Can Strata's CUDA split engine run one Flash-Next model across two DGX Sparks? The existing split configuration expects the GPUs to be visible on one host. With two Sparks, I want both machines to hold and compute on their share of the weights while serving the same request. I have a prototype that runs one MPI process per host. MPI carries commands and host results, NCCL carries GPU tensors, and the server uses the existing engine protocol. I tested the candidate on two GB10s over their direct link. The Unsloth UD-IQ4_XS model I used has Q5_0 routed down weights, which the current split loader rejects. The proposed build handles those weights as well. With the same Unsloth UD-IQ4_XS model and DFlash2 settings, median decode speed went from 76.93 to 100.00 tok/s for a Python coding prompt and from 59.33 to 72.27 tok/s for an English LRU-cache explanation. Each configuration had four measured 512-token replies per workload. Cold prefill for a 33,280-token English document took 18.39 s on one Spark and 14.86 s on two, with two runs per configuration. All four long-prompt runs retrieved the three requested codes. Both short prompts produced different continuations across rank counts. Would an opt-in Linux CUDA build with that scope fit? Draft PR #6263 contains the implementation and validation details. ### What do you do today instead? I run the Spark build on one host or use a local build of the cluster prototype. The broader multi-node question came up in #1750.
Related on this site
Quick links to related pages.