#6141 [Engine] Scope proposal for NVLink expert uploads on asymmetric PCIe links
open · @tubsGreatVictory · 3 commentaires · Sur GitHub
BenchmarksSetup & installMulti-GPUNVIDIA / CUDALinux
Description
### Area Engine ### What do you want to do? Could Strata support an opt-in Linux CUDA route that uploads streamed prefill expert weights over PCIe Gen3 x16 to CUDA1, then forwards them over NVLink to CUDA0 on a Gen3 x4 link? The prototype uses 8 MiB of staging space and checks for two RTX 2080 Ti cards, bidirectional NVLink, and CUDA peer access. #5999 covers P2P transfers for the layer-split prompt helper. In this proposal CUDA1 stages the weights and CUDA0 computes with them during eligible solo prefill. The prototype defaults to off. ### Evidence On my dual RTX 2080 Ti setup, paired 128K runs increased median prefill throughput from 1,103.5 to 1,888.6 tok/s and reduced median time to first token from 118.90 to 69.51 seconds. Draft PR #6131 has the patch and validation details. Would you prefer a separate opt-in route, shared transport code with #5999, or a smaller first step?
Sur le site
Liens install, modèles, releases.