#6514 [Benchmark] RX 7600 XT (gfx1102): ROCm 7.2.4 vs 10.1, and the CachyOS ROCm 10.1 crash
open · @DamirLukina88 · 0 comentários · No GitHub
BenchmarksServer & APIAMD / HIPModels & quantsDocumentationWindowsLinux
Descrição
## Problem and result
On CachyOS, the update from ROCm 7.2.4 to 10.1 (`hip-runtime-amd` 10.1-2.1 from `cachyos-extra-v3`) makes the engine exit at every start with SIGSEGV inside `libamdhip64.so.7`, at its first `hipStreamCreateWithFlags` ("the engine exited before it was ready"). It's a HIP runtime packaging bug, not an engine bug. The docs don't mention it yet, and there's no measurement of gfx1102 on ROCm 10.1.
This PR adds, results and docs only (no engine changes):
- `docs/bench-results/2026-11-21-community-rx-7600-xt-rocm-10.1/`: a community benchmark for an RX 7600 XT 16 GB (gfx1102). The standard `strata_bench.py --long --clients 4` suite ran once on ROCm 7.2.4 and once on ROCm 10.1 with Arch's `libamdhip64`, with the same engine binary and model. It includes the per-run READMEs and `results.json` written by the tool, a 23-line crash reproducer (`repro/narrow.hip`) and an alignment shim (`repro/align64_new.cpp`) that confirms the cause.
- `docs/AMD_HIP.md`: a new section, "Linux: ROCm 10.1 on CachyOS crashes at start (x86-64-v3 builds)", with the cause and workarounds.
- `docs/COMMUNITY_BENCHMARKS.md`: one entry in the community list.
Results (medians):
| Test | ROCm 7.2.4 | ROCm 10.1 + Arch libamdhip64 |
| --- | ---: | ---: |
| Short chat decode | 29.8 tok/s | 29.2 tok/s |
| 4K / 16K / 32K prompt | 339.0 / 365.4 / 363.3 tok/s | 344.3 / 371.5 / 368.3 tok/s |
| Agent session decode | 32.9 tok/s | 33.7 tok/s |
| 4 clients total | 22.4 tok/s | 23.3 tok/s |
Every difference is between −2% and +4%. The two runs also differ in kernel and GPU voltage offset (see Limits), so the conclusion is: **ROCm 10.1 with Arch's `libamdhip64` is at least as fast as 7.2.4 on this card**.
## Related discussion
- Upstream HIP bug: ROCm/rocm-systems#14003 and #11256; fix in ROCm/rocm-systems#16099, which is still open.
- CachyOS packaging: CachyOS/CachyOS-PKGBUILDS#2143 (v3 build) and #2137 (znver4).
- Strata: none. This is a docs and results submission, done the way COMMUNITY_BENCHMARKS.md describes.
## Approach
- **Cause.** `roc::VirtualGPU` is `alignas(64)`, but `amd::ReferenceCountedObject` has a class `operator new` without a `std::align_val_t` overload, so the object gets 16-byte aligned memory. The v3 build's constructor stores with `vmovdqa` (32-byte aligned) and faults. Arch's generic x86-64 build doesn't.
- **The docs point to existing fixes and don't change any defaults:**
- Arch's `extra/hip-runtime-amd` plus `IgnorePkg`
- Arch's `opt/rocm/lib` placed first in `lib_dirs`
- staying on 7.2.4
- **The results folder follows the report template** in COMMUNITY_BENCHMARKS.md. The tool's own per-run output is kept unchanged in subfolders.
## Validation
Environment:
- **Hardware:** RX 7600 XT 16 GB (gfx1102, XFX SWFT 210) at PCIe 4.0 x8; Ryzen 5 3600 (Zen 2, AVX2); 32 GB DDR4-3600; ASUS ROG Strix B550-F, BIOS 3646.
- **OS and power:** CachyOS; LACT at 150 W cap, 2500 MHz max clock, COMPUTE profile.
- **Engine:** `0.1.44.1+0.2.12-fixes`, a HIP source build for gfx1102. It was built on 2026-11-07 from a fork (DamirLukina88/D.L.-AI---Strata at `71a5118d`, upstream 0.2.12-0.2.17 server fixes ported), and the same binary was used in every run. Model `qwen3.8-flash-next-coder-iq1_m`, 65K context, 1 slot, `--vram-reserve-mib 3072`.
**Passed:**
- **ROCm 7.2.4 benchmark.** Kernel 7.3.4-1-cachyos, ROCm 7.2.4 (a private copy of the 7.2.4 packages via `lib_dirs`), −50 mV, 2026-11-20 14:38-14:51. Command: `python tools/strata_bench.py --long --clients 4` against the running server. The full suite completed in 768 s with 0 request errors and 0 amdgpu/AER/MCE kernel messages. Output: `rocm-7.2.4/`.
- **ROCm 10.1 benchmark.** Same command on kernel 7.3.5-2-cachyos, CachyOS ROCm 10.1 with `libamdhip64.so.7.16.0` from Arch's `extra/hip-runtime-amd` 10.1-2 (signature checked) first in `lib_dirs`, stock voltage, 2026-11-21 18:02-18:14. The full suite completed in 731 s with 0 request errors and 0 kernel messages. Output: `rocm-10.1-arch-hip/`.
- **32K-only retest on ROCm 10.1 + Arch HIP:** 3/3 prompts passed, median 367.1 tok/s.
- **Reproducer.** Build: `hipcc -O2 --offload-arch=gfx1102 -o narrow repro/narrow.hip`.
- With Arch's `libamdhip64` 10.1-2 (`LD_LIBRARY_PATH`): `./narrow mS` printed `OK` 3/3, and `MS`, `CS`, `HCS`, `HMCS` also passed.
- On ROCm 7.2.4: `mS`, `MS`, `CS` printed `OK`.
- **Shim.** Build: `g++ -O2 -shared -fPIC -o align64_new.so repro/align64_new.cpp`. With `LD_PRELOAD=./align64_new.so` and the unmodified CachyOS library, `mS` passed 5/5 and `CS` and `HMCS` passed.
- **Links:** every relative link in the new and changed docs resolves to an existing file or heading in this branch.
**Failed:**
- **The bug itself.** With CachyOS `hip-runtime-amd` 10.1-2.1, `./narrow mS` gave SIGSEGV 3/3 (top frame `libamdhip64.so.7 + 0x74d5b8`, `vmovdqa %ymm1,0xc0(%rdi)` with `rdi` 16 mod 64), and so did `MS`, `CS`, `HCS`, `HMCS` and `hipMemcpyAsync` from `malloc` memory. The engine crashed at every start, the same way, on kernels 7.3.4 and 7.3.5.
- **Things tried that didn't help:**
- an engine rebuilt against 10.1
- `HSA_ENABLE_SDMA=0`, `GPU_MAX_HW_QUEUES=1`, `HSA_OVERRIDE_GFX_VERSION` set or unset
- swapping only Arch's `libhsa-runtime64`
- **One stall on 10.1 + Arch HIP.** The first full suite (2026-11-21) passed everything before its first 32K prompt, then stopped there while reading layer 45 of the first chunk. The engine watchdog ended it (exit −6) at 17:49:06, after 284 s without progress.
- During the stall the GPU was idle and there were no kernel messages.
- The stall report showed 6.3 GiB RAM available, about 578K major page faults, and a thread in `svm_range_set_attr`.
- It didn't happen again in the next 6 32K prompts.
- Evidence: `rocm-10.1-arch-hip/stalled-run/` (suite output, engine stall report, GPU sensor log) and `retest-32k/`. The tool wrote no `results.json` for the aborted suite.
**Not run:**
- **Upstream `main` engine.** The benchmarks used the fork's 0.1.44.1-based engine, not this branch's.
- **Repetition:** no repeated or interleaved A/B; one suite per runtime.
- **Correctness:** no answer-quality checks (recall, HumanEval).
- **Controlled stall test:** the 32K stall wasn't tested against the host-memory reclaim workarounds in AMD_HIP.md.
- **Other setups:** other GPUs and arches, znver4 builds, Windows, and ROCm 10.2.
## Limits
- The two runs differ in kernel (7.3.4 → 7.3.5) and GPU voltage offset (−50 mV → stock). The 10.1 run drew more power (128 vs 117 W average), so the small speed differences can't be credited to ROCm alone.
- The 32K stall's cause is unknown. It fits the `--mmap-experts` case in AMD_HIP.md (#920: RX 6800, 31 GiB RAM). Experts are mmap-ed here too, on 31 GiB. That case's `GPU_PINNED_MIN_XFER_SIZE=1048576` workaround wasn't tried.
- The workarounds are only needed until CachyOS ships a `hip-runtime-amd` with ROCm/rocm-systems#16099, or a generic build. `./narrow mS` passing is only a smoke check: confirm the package carries #16099 before dropping the workaround.
- Launch settings: the bench tool records no launch config when it runs against an already-running server. The sanitized server config of each run is in `launch/`; the two differ only in `lib_dirs`, and the speed-projection control vector (scale 1.0, layers 4-44) was on in both runs.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
No site
Links install, modelos, releases.