#5781 [Engine] Support N+ dedicated MoE expert-cache helper GPUs
open · @MarsLuay · 1 comments · View on GitHub
NVIDIA / CUDAModels & quantsDocumentation
Description
This request pairs with #5780, which proposes experimental Tesla M10 support. Strata currently limits dedicated expert-cache helpers to three GPUs through `--expert-cache-device1..3`. This is restrictive for hardware such as the Tesla M10, where each 32GB card contains four independent 8GB GPUs. Rather than increasing the limit to another fixed number, I propose supporting a **dynamic number (N) of expert-cache helper GPUs**, determined by the available hardware and resources. ### Area [Engine] ### What do you want to do? Extend Strata's existing expert-cache system to support **N helper GPUs instead of a hard-coded limit of three**. **Example configuration:** - RTX 3080 Ti 12GB + Tesla V100 32GB: main inference - 2× Tesla M10: eight independent expert-only GPUs (64GB total) - Qwen3.8-Flash-Next IQ3_S, 256K context - 64GB system RAM Tesla M10 compatibility is addressed separately in #5780. This request should support any compatible GPU architecture. **Requested functionality:** - Replace the fixed three-helper limit with a dynamically sized GPU device list. - Automatically detect available GPU devices and their usable VRAM. - Distribute experts across N GPUs based on VRAM capacity, routing frequency, GPU performance, and PCIe bandwidth. - Allow GPUs to be dedicated to expert storage and computation without running dense layers, attention, or KV cache. - Avoid assigning work to slower helpers when doing so reduces performance. - Share resident experts across concurrent conversations. - Preserve existing layer splitting, RAM fallback, and backward compatibility with `--expert-cache-device1..3`. - Report per-GPU expert allocations, utilization, and performance. The number of helpers should be limited by actual hardware and runtime constraints rather than an arbitrary fixed count. Existing behavior should remain unchanged unless additional helpers are explicitly enabled. ### What do you do today instead? Use up to three expert-cache helpers, leaving additional GPUs unused for this purpose, or maintain a custom implementation. ### Related issues - [[#5780](https://github.com/Niko1221/Strata/issues/5780)](https://github.com/Niko1221/Strata/issues/5780): Experimental Tesla M10 (`sm_50`) expert-cache support. This issue would allow all eight M10 GPU dies to be used. - [[#4145](https://github.com/Niko1221/Strata/issues/4145)](https://github.com/Niko1221/Strata/issues/4145): Demonstrates an M10 expert tier using four GPU dies. Closed because community benchmarks were moving to a dedicated website, not because the approach was rejected. - [[#4948](https://github.com/Niko1221/Strata/issues/4948)](https://github.com/Niko1221/Strata/issues/4948): Open memory-manager redesign proposing an "experts on any card" architecture. Implementation should avoid duplicating this work. - [[#4929](https://github.com/Niko1221/Strata/issues/4929)](https://github.com/Niko1221/Strata/issues/4929): Open issue concerning expert-helper prefill performance with concurrent conversations. - [[docs/SECOND_GPU.md](https://github.com/Niko1221/Strata/blob/main/docs/SECOND_GPU.md)](https://github.com/Niko1221/Strata/blob/main/docs/SECOND_GPU.md): Documents the existing three-helper limitation. ### Validation status Feature request only; no benchmarks with more than three helpers have been performed for this request. Suggested testing: configurations with 1, 3, 4, 8, and additional available helper GPUs; memory limits; output correctness; prefill/decode performance; and concurrent throughput. Would maintainers prefer extending the existing expert-cache system to support N devices, or incorporating this into the memory-manager work in #4948?
Related on this site
Quick links to related pages.