#6562 [Engine] hy4: opt-in HIP build of the multi-GPU expert split
open · draft · @orkun-soylu · 0 评论 · 去 GitHub 看
BenchmarksMulti-GPUAMD / HIPNVIDIA / CUDAModels & quantsDocumentation
说明
## Problem and result On HIP, `strata-hy4 --gpus 0,1` puts no experts on the second card: the peer-GPU expert split is compiled only under `#if HY4_CUDA` (four blocks in `hy4_gpu_model.hip.h` and `hy4_graph.hip.h`), and with peers present the adaptive VRAM cache is also off. On a W7800 48 GB + RX 7900 XTX 24 GB the XTX sat idle and the experts that did not fit the W7800 ran on the CPU. With `-DHY4_HIP_PEERS=1` those blocks build on HIP. Same machine, Hy4-Flash SQ-96, 128K context: decode 30.7 → 36.5 tok/s, prompt reading 817 → 1,241 tok/s at 24K tokens (this commit, without and with the flag). Without the flag a HIP build is unchanged. ## Related discussion none. The change is opt-in at build time and leaves the default path alone, so I went straight to a PR; happy to move it to an issue first if you prefer. ## Approach The peer code is spelled with `cudaXxx` names, and each one has a HIP twin with the same signature. `hy4_gpu.h` already maps `hipXxx` → `cudaXxx` for CUDA; this adds the reverse map for those 14 names in its HIP branch, behind `HY4_HIP_PEERS` (default 0). The four guards become `#if HY4_CUDA || HY4_HIP_PEERS`, so the CUDA build compiles the same blocks as before. `HY4_PEER_RESERVE_MB` (optional) overrides the 960 MiB each peer keeps free. 960 is sized for 4,096-token prompt chunks; with 8,192 the XTX peer ran out of memory on the first long prompt, 2048 works. Unset or not a positive number, it stays 960. On CUDA this is the one change to compiled code: a `getenv` at load time, with the same value when the variable is unset. The output with peers is deterministic run to run but not bitwise equal to one card, because each card adds its own partial sums (the same is true of the CUDA split). `docs/AMD_HIP.md` gets a short section with the flag, the reserve and the numbers. ## Validation Environment: Ubuntu 26.04, Radeon PRO W7800 48 GB (main, `gpu` [1, 0]) + RX 7900 XTX 24 GB, Ryzen 5 7600, 64 GB DDR5; Hy4-Flash SQ-96 + Engram-Q8 + Vision, `--max-context 131072`, DFly2 drafting. Base revision 6e38f588a (v0.2.18). Built with ROCm 10.1's clang (`-x hip --offload-arch=gfx1100 -DHY4_MMA=1`), source list as `engine/cmake/backends/quant.cmake` and `build_hy4_sycl.sh`, run on the ROCm 7 runtime libraries of the `hip-rx7900` package. Benchmark: 4 decode prompts at temperature 0, thinking off, and two prompts of 6,425 and 24,432 tokens; one round each. **Passed:** - HIP default path unchanged: `hy4_main.hip.cpp` preprocessed (`-E -P`, host and device passes) at 6e38f588a and at this PR without the flag differ only in the `__LINE__` values of the error macro in `hy4_gpu_model.hip.h` (two added lines). - Without the flag, W7800 alone, two runs: decode 30.8 / 30.6 tok/s, prompts 813 / 812 (6K) and 816 / 818 tok/s (24K). Unmodified 6e38f588a built the same way, two runs: 29.5 / 29.3, 813 / 810, 816 / 817. - With `-DHY4_HIP_PEERS=1`, W7800 + XTX, `HY4_PEER_RESERVE_MB=2048`: decode 36.5 tok/s, prompts 1,205 (6K) and 1,241 tok/s (24K). Decode and prompt output hashes identical to an earlier revision of this patch measured the day before (decode 37.1, 1,194 / 1,214); with the split the output repeats run to run. - With the flag, 10 hard reasoning problems × 2 at effort medium: 20/20, all finished with `stop` (an earlier revision of this patch with the same four blocks; not re-run on this commit). - Pictures with the split on: one image answered in 3.3 s with `HY4_RESERVE_GB=8.8` (earlier revision, same four blocks). **Failed:** - With the flag and the default 960 MiB reserve: the XTX peer ran out of memory at 8,192-token chunks (the reason for `HY4_PEER_RESERVE_MB`). - With the split on and the default `HY4_RESERVE_GB`, the first image failed with `cannot allocate 890 MiB for the vision weights` (0.6 GiB left on the W7800). That is a sizing question rather than a bug in this patch; `HY4_RESERVE_GB=8.8` fixed it. Not changed here. **Not run:** - CUDA build: I have no CUDA toolchain at hand for this revision. The CUDA side changes only the `HY4_PEER_RESERVE_MB` lookup; a CUDA build and a two-GPU CUDA run would confirm it. - SYCL and Metal: not touched. - Other AMD cards than gfx1100, and peers without P2P access (`hipDeviceCanAccessPeer` is checked by the existing code; I only have a pair that has it). ## Limits - On one card my own builds do not repeat their output run to run at temperature 0 (unmodified 6e38f588a: two runs, two different decode hashes), so I could not use output hashes to show the flag-off build matches; the preprocessed source above is the evidence for that. - The repo has no build recipe for `strata-hy4` on HIP that I could find, so the flag is documented but not wired into a package build. - The reserve is a manual knob; sizing it from the prefill chunk would be cleaner, but I did not want to change the CUDA default.
本站相关内容
相关页面的快捷入口。