#6141 [Engine] Scope proposal for NVLink expert uploads on asymmetric PCIe links

open · @tubsGreatVictory · 3 コメント · GitHub で見る

BenchmarksSetup & installMulti-GPUNVIDIA / CUDALinux

本文

### Area

Engine

### What do you want to do?

Could Strata support an opt-in Linux CUDA route that uploads streamed prefill expert weights over PCIe Gen3 x16 to CUDA1, then forwards them over NVLink to CUDA0 on a Gen3 x4 link? The prototype uses 8 MiB of staging space and checks for two RTX 2080 Ti cards, bidirectional NVLink, and CUDA peer access.

#5999 covers P2P transfers for the layer-split prompt helper. In this proposal CUDA1 stages the weights and CUDA0 computes with them during eligible solo prefill. The prototype defaults to off.

### Evidence

On my dual RTX 2080 Ti setup, paired 128K runs increased median prefill throughput from 1,103.5 to 1,888.6 tok/s and reduced median time to first token from 118.90 to 69.51 seconds. Draft PR #6131 has the patch and validation details.

Would you prefer a separate opt-in route, shared transport code with #5999, or a smaller first step?

関連リンク

インストール・モデル・リリースへのサイト内リンク。