#6322 [Engine] v0.2.16 source build on gfx1201 segfaults before any output with system ROCm 7.16; STRATA_HIP_QUEUE_PRIME=0 and the official rx9070 package are unaffected

open · @Thxeverybody · 4 コメント · GitHub で見る

Setup & installServer & APIMulti-GPUAMD / HIPNVIDIA / CUDAModels & quantsSecurityDocumentationWindowsLinux

本文

### Area

[Engine]  (speed, kernels, models, the server and API)

### Desktop app, Terminal or source?

Source (START-HERE.bat, setup.sh or a build)

### What happened

I built the engine from tag v0.2.16 with the release recipe of `tools/packaging/build_hip_linux.sh` (minus the packaging/library bundling), linking against the system ROCm 7.16. Running the resulting binary with model arguments **segfaults before it prints a single line** — 8 of 8 runs, exit 139, empty log. The first output line of a healthy start (`strata generate: opened 2 GPU queues in a fixed order …`) never appears in a crashing run.

The **official `strata-engine-0.1.62-linux-x86_64-hip-rx9070` package** (sha256-verified, bundled ROCm libs, `RUNPATH $ORIGIN`) runs the same arguments cleanly — 3 of 3 — and a full IQ3_S `--serve` load with it succeeds (4883 expert slots / 9.5 GiB, MTP draft, control vector, PLE, WMMA all active; it serves normally afterwards).

Boundary I found: **`STRATA_HIP_QUEUE_PRIME=0` lets the source build start normally** (banner prints, clean usage exit). Under gdb the same argv also survives (ptrace timing), so I have no crash backtrace yet — the fault site is inferred, not proven.

This points at `prime_hw_queues()` (`engine/src/program/generate.cpp`, ~line 3391 at this tag): the early HIP sequence (`cudaGetDeviceCount` → `cudaGetDeviceProperties` → `cudaMalloc(256)` → `hipDeviceGetStreamPriorityRange` → `hipStreamCreateWithPriority` / `cudaStreamCreateWithFlags` → `cudaMemsetAsync` → `cudaStreamSynchronize`) that runs on the main thread before any loader thread exists. The release packages are built against TheRock wheels (docs/AMD_HIP.md: Linux rx9070 → 7.10.0a; the package's BUILD.json reports hipblaslt_version 100300), while my source build links system ROCm 7.16.

One caveat I cannot rule out: the package's BUILD.json names source commit `18526b879` (f-amd-vision), which differs from the tag commit `64524fb9f` I built. The `QUEUE_PRIME=0` boundary points at the prime path either way, but I have not bisected code-vs-ROCm.

**What I observed (all on the same machine, same day):**

| Binary | Environment | Result |
|---|---|---|
| source build (tag v0.2.16) | default | SIGSEGV **8/8**, zero output, exit 139 |
| source build | `STRATA_HIP_QUEUE_PRIME=0` | starts, prints banner, clean usage exit |
| source build | default, under gdb | survives, prints banner, clean usage exit |
| official 0.1.62 rx9070 package | default, bundled libs (no `LD_LIBRARY_PATH`) | **3/3** clean, "opened 2 GPU queues" prints |
| official package | full IQ3_S `--serve` load | loads completely (4883 slots, MTP, cvec, PLE, WMMA), serves |

A single `--help` invocation of the source build also survived once — the crash is timing-sensitive, but with `--pack` it was 8/8.

**Not the same as #5152:** that one is a 4-card layer-split crash on the first chat request with the official packages, and it crashes with `STRATA_HIP_QUEUE_PRIME=0` too. This one is single-card, at startup before any output, only with the source build against system ROCm 7.16, and `=0` avoids it.

**Workaround:** use the official engine package (what I did — it serves fine), or `STRATA_HIP_QUEUE_PRIME=0` for a source build. The latter loses the deterministic queue placement the prime exists for (per the comment in generate.cpp: without it, about half of starts land the two busy queues on the same pipe, 4K prompts ~13% slower).

Happy to provide a core dump / backtrace (needs an idle-GPU window) or to test a patch.

### Smallest steps or command that reproduce it

Needs an **idle GPU** (with VRAM full, the prime's `cudaMalloc(256)` fails and the function returns early — no crash).

```bash
git clone --branch v0.2.16 https://github.com/Niko1221/Strata && cd Strata
cmake -G Ninja -S engine -B build-hip -DCMAKE_BUILD_TYPE=Release \
  -DSTRATA_ENABLE_HIP=ON -DSTRATA_ENABLE_CUDA=OFF -DSTRATA_BUILD_TESTS=OFF \
  -DCMAKE_HIP_ARCHITECTURES=gfx1201 \
  -DCMAKE_HIP_COMPILER=/opt/rocm/lib/llvm/bin/clang++ \
  -DCMAKE_HIP_COMPILER_ROCM_ROOT=/opt/rocm -DCMAKE_PREFIX_PATH=/opt/rocm \
  "-DCMAKE_HIP_FLAGS=--rocm-path=/opt/rocm --rocm-device-lib-path=/opt/rocm/amdgcn/bitcode" \
  -DSTRATA_PORTABLE=ON
cmake --build build-hip --target strata -j8

./build-hip/strata --pack <any-pack-dir>
# → SIGSEGV before any output, 8/8 runs (exit 139)

STRATA_HIP_QUEUE_PRIME=0 ./build-hip/strata --pack <any-pack-dir>
# → banner prints, exits with the normal "--tokens is required" usage
```

**Secondary note for source builders on Arch/CachyOS:** `tools/packaging/build_hip_linux.sh` defaults to `--rocm-device-lib-path=$HP/lib/llvm/amdgcn/bitcode` and `$HP/llvm/bin/clang++`. On this distro's ROCm the device libs live at `$HP/amdgcn/bitcode` and the compiler at `$HP/lib/llvm/bin/clang++`; with the script's defaults every HIP compile fails with `cannot find ROCm device library` (configure still succeeds). Related: #4512.

### Reproduction status

Reproduced locally

### Strata version

Strata 0.2.16 · engine 0.1.62 (tag v0.2.16, commit 64524fb9f; official package built from 18526b879)

### Graphics card, memory and system

RX 9070 XT 16 GB (gfx1201), desktop on the iGPU (Radeon 610M) · Ryzen 7 9700X · 128 GB RAM · CachyOS (Arch-based), kernel 7.3.5-1-cachyos · system ROCm 7.16.0 (hipcc 7.16.0-4f6d503, AMD clang 24.0.0git), hipBLASLt 1.4.1 · cmake 4.5.0, ninja

### Model, size and settings

Qwen3.8-Flash-Next GSQ-RCO IQ3_S · 262144 context · pictures on. The crash happens before any model handling — `--pack <dir>` alone reproduces it.

### Logs

The crashing runs produce no engine output at all (empty log, exit 139). Attaching two files from the working side: the gdb run of the same source-build argv (survives, prints the full banner — shows how far a healthy start gets) and the full IQ3_S `--serve` load log of the official package (loads completely, serves). Both contain only public model names and local paths.

** This issue was drafted with support of Qwen 4 max on Hermes **

[gdb-crash.log](https://github.com/user-attachments/files/35107338/gdb-crash.log)

[loadtest-pkg.log](https://github.com/user-attachments/files/35107346/loadtest-pkg.log)

関連リンク

インストール・モデル・リリースへのサイト内リンク。