#6599 [Engine] strata-hy4: dual-GPU expert-parallel support (--gpus 2)

open · @fogliaye91-cmd · 0 comments · View on GitHub

BenchmarksMulti-GPUAMD / HIPModels & quants

Description

### Feature request

Add multi-GPU (dual-GPU expert parallel) support to the `strata-hy4` engine, similar to the `--gpus 2` / dual-GPU expert-parallel (ep_du) mechanism that already exists for `strata-inkling`.

### Motivation

I run Hy4-Flash (SQ-64, 220B total / 12B active) on 2x Intel Arc Pro B65 (Battlemage, 32 GiB VRAM each) with Strata 0.1.65 engine / v0.2.19 server on Ubuntu 26.04.

Current single-GPU placement (`hy4_fit.place`):

- vram_experts: 17.84 GiB
- ram_experts: 38.20 GB (resident in RAM, random-read on every decode pass)

The decode bottleneck is exactly the RAM-resident experts: with everything else optimal (DFlash2 drafter enabled, ~17% acceptance, pinned memory), decode sits at ~8-9 tok/s. Two more cards' worth of VRAM sits idle.

If `strata-hy4` supported `--gpus 2` with dual-GPU expert placement (63.78 GiB combined VRAM on this box), the large majority of experts could be held in VRAM and decode speed would improve substantially. Since `ep_du` already exists for the inkling engine family (#6586 touches `strata-inkling --gpus 2`), I hope the same mechanism can be extended to hy4.

I confirmed the 0.1.65 `strata-hy4` binary exposes no multi-GPU flags (`--gpus`, `--peer-device`, layer-split are all absent; only `--gpu` and `--gpu-experts` exist), so this is not a configuration issue on my side.

Context: I also filed #6572 (hy4 garbage output on Battlemage, fixed by v0.2.19 XMX dispatch fix). Happy to test any WIP build — dual Arc B65 hardware is available.

### Environment

- Engine: strata-hy4 0.1.65 (bmg / SYCL AOT)
- Server: Strata v0.2.19
- GPU: 2x Intel Arc Pro B65 (Battlemage bmg-g31), 32 GiB each
- CPU: AMD EPYC 7302, 128 GB RAM
- OS: Ubuntu (kernel 6.17), level_zero backend

Related on this site

Quick links to related pages.