#6513 [Engine] Linux HIP: engine crashes at start (SIGSEGV in libamdhip64) with CachyOS's hip-runtime-amd 10.1 v3/znver4 builds; Arch's build works
open · @DamirLukina88 · 0 commentaires · Sur GitHub
BenchmarksServer & APIAMD / HIPModels & quantsLinux
Description
### Area
[Engine]
### Desktop app, Terminal or source?
Source (a build)
### What happened
On CachyOS, the ROCm 10.1 update (2026-11-16) makes every HIP engine start fail. The server reports that the engine exited before it was ready, and the engine dies with SIGSEGV inside `libamdhip64.so.7` the first time the HIP runtime creates its device queue. That is either the first `hipMemcpyAsync` in `probe_pcie_h2d_gbps` or the first `hipStreamCreateWithFlags`, depending on heap layout:
```
Signal: 11 (SEGV) si_code: SI_KERNEL
#0 libamdhip64.so.7 + 0x74d5b8
#1 libamdhip64.so.7 + 0x7291af
#2 libamdhip64.so.7 + 0x70e7dc
#3 libamdhip64.so.7 + 0x545a34
#4 libamdhip64.so.7 + 0x54eb54
#5 hipStreamCreateWithFlags (libamdhip64.so.7 + 0x628874)
#6 main (strata)
```
The bug is not in Strata. CachyOS rebuilds Arch's `hip-runtime-amd` with `-march=x86-64-v3` (and znver4). That exposes an upstream HIP runtime bug: `amd::ReferenceCountedObject` has a class-scope `operator new(size_t)` and no `std::align_val_t` overload, so `new roc::VirtualGPU` (64-byte aligned members) gets only 16-byte heap alignment, and the AVX build's aligned `vmovdqa ymm` stores fault. Arch's generic build uses 16-byte stores and does not crash.
- CachyOS: https://github.com/CachyOS/CachyOS-PKGBUILDS/issues/2143 (v3, gfx1201 / gfx1032 / gfx1030 / gfx1102; my confirmation with the faulting instruction is there) and #2137 (znver4)
- AMD: https://github.com/ROCm/rocm-systems/issues/14003, https://github.com/ROCm/rocm-systems/issues/11256; fix in https://github.com/ROCm/rocm-systems/pull/16099 (open as of 2026-11-21, so ROCm 10.2.0 does not have it)
Another commenter on #2143 hit the same crash with Strata 0.2.15 / engine 0.1.61 on an RX 6600 XT, so it is not specific to my build or engine version.
I'm filing this so other Linux users who search for the crash find the cause and a workaround. Rebuilding the engine does not help: an engine built against ROCm 10.1 crashes the same way.
### Workarounds (both tested)
1. System-wide: `sudo pacman -S extra/hip-runtime-amd`, then `IgnorePkg = hip-runtime-amd` in `/etc/pacman.conf` until CachyOS ships a fixed build.
2. Without root, using the model config's `lib_dirs`: unpack Arch's package and put its library folder first.
```
curl -O https://fastly.mirror.pkgbuild.com/extra/os/x86_64/hip-runtime-amd-10.1-2-x86_64.pkg.tar.zst
mkdir -p rocm-10.1-arch && bsdtar -xf hip-runtime-amd-10.1-2-x86_64.pkg.tar.zst -C rocm-10.1-arch
```
```json
"lib_dirs": ["<folder>/rocm-10.1-arch/opt/rocm/lib", "/opt/rocm/lib", "/opt/rocm/lib/llvm/lib"]
```
Only `libamdhip64` has to come from Arch. The rest of ROCm stays the system's 10.1.
Before I found the cause, I kept the engine running by pointing `lib_dirs` at an unpacked copy of the cached 7.2.4 packages. That also works.
### Speed: ROCm 10.1 (Arch's libamdhip64) vs 7.2.4, same engine binary
`./bench.sh --url http://127.0.0.1:8080 --long --clients 4` against the running server, `--compare` output. The 7.2.4 run is from 2026-11-20, and the 10.1 run is from 2026-11-21 (the second full run that day). Same engine binary, model, config and power settings (150 W, 2500 MHz). The one difference: the 7.2.4 run had a -50 mV GPU undervolt and the 10.1 run had none.
| | 7.2.4 | 10.1 | change |
|---|---:|---:|---:|
| chat decode tok/s | 29.80 | 29.20 | -2.0% (range 25.4-43.6) |
| chat first token s | 1.49 | 1.44 | -3.4% |
| 4K prompt tok/s | 339.00 | 344.30 | +1.6% |
| 16K prompt tok/s | 365.40 | 371.50 | +1.7% |
| 32K prompt tok/s | 363.30 | 368.30 | +1.4% |
| 32K first token s | 88.28 | 87.09 | -1.3% |
| agent turn first token s | 9.88 | 10.04 | +1.6% |
| agent decode tok/s | 32.90 | 33.70 | +2.4% |
| 4 clients total tok/s | 22.41 | 23.32 | +4.1% |
So the workaround costs no speed on this card. The differences are within a few percent, and the undervolt difference means I can't attribute them to ROCm.
One observation I can't explain yet. In the first full benchmark on ROCm 10.1, the first 32K prompt stopped progressing after its first chunk. The engine's no-progress watchdog caught it ("reading the prompt (batched): layer 45 of the prompt chunk from token 0", 0 layers served for 284 s), stopped the engine, and the server restarted it as designed. During the stall the GPU was idle (2-4 % busy, 24 W) and the kernel logged no amdgpu errors. It did not come back in the next 6 prompts of 32K: a 32K-only retest passed 3/3, and so did the full rerun. If it happens again I'll open a separate issue with the thread states.
### Smallest steps or command that reproduce it
On CachyOS with `hip-runtime-amd 10.1-2.1` from `cachyos-extra-v3`, start any HIP engine, or run this:
```cpp
// hipcc -O2 --offload-arch=gfx1102 -o t t.hip && ./t -> SIGSEGV (3/3)
#include <hip/hip_runtime.h>
int main() {
void* d; hipMalloc(&d, 4096);
hipStream_t s; return hipStreamCreateWithFlags(&s, hipStreamNonBlocking);
}
```
### Reproduction status
Reproduced locally
### Strata version
A fork based on 0.1.44.1 with later fixes ported. The engine says `0.1.44.1` (built from source, HIP, gfx1102). Also reported on #2143 with Strata 0.2.15 / engine 0.1.61.
### Graphics card, memory and system
AMD Radeon RX 7600 XT 16 GB (gfx1102), Ryzen 5 3600, 32 GB DDR4, CachyOS (kernel 7.3.5), ROCm 10.1
### Model, size and settings
Qwen3.8-Flash-Next Coder IQ1_M, 64K context, int8 KV cache, `--spec 4` with MTP, pictures on
Sur le site
Liens install, modèles, releases.