#5559 [Engine] 0.1.58 hip-rx9070 (gfx1201): strata-host crashes repeatedly (GPF in libhsa-runtime64, exit -6/-11) under 35K-92K prompts; 0.1.57 runs clean

open · @avocadori · 7 コメント · GitHub で見る

BenchmarksSetup & installServer & APIAMD / HIPModels & quantsSecurityLinux

本文

### Area
[Engine]

### Desktop app, Terminal or source?
Terminal (`strata serve --config <run config> --host 0.0.0.0 --port 8080 --parallel 2`, run as a systemd user unit)

### What happened
After the daily update installed engine 0.1.58 (05:15 JST, 2026-11-08; app 0.2.11 -> 0.2.12, control and engine 0.1.57 -> 0.1.58), the engine process (`strata-host`) crashed 9 times in about 5 hours under our normal traffic. Engine 0.1.57 had run the same traffic since 2026-11-07 06:20 without a single crash in the kernel log, and after switching back with `strata components use 0.1.57` it has been clean again.

Crash times (JST) and what the kernel / control logged:

```
06:31:33 kernel: traps: strata-host[103589] general protection fault ip:7d0999ed5a65 sp:7ffc6a9903b0 error:0 in libhsa-runtime64.so.1[d4a65,7d0999e9c000+137000]
07:36:28 kernel: traps: strata-host[105216] general protection fault ip:75a8f58d5a65 ... in libhsa-runtime64.so.1[d4a65,...]
09:45:04 kernel: traps: strata-host[172572] general protection fault ip:7d47cf6d5a68 ... in libhsa-runtime64.so.1[d4a68,...]
10:02:23 control: the engine stopped unexpectedly (exit code -6). The usual cause is running out of RAM ...   (no OOM-killer line in the kernel log)
10:11:30 control: the engine stopped unexpectedly (exit code -6). The engine stopped itself because it had stopped making progress - a hang it caught.
10:14:52 kernel: traps: strata-adapt[193132] general protection fault ip:72bd01cd5a68 ... in libhsa-runtime64.so.1[d4a68,...]
10:14:54 control: the engine stopped unexpectedly (exit code -11). The engine stopped itself because it had stopped making progress - a hang it caught.
10:16:19 control: the engine stopped unexpectedly (exit code -6). ... a hang it caught.
10:18:24 kernel: strata-host[194977]: segfault at 7f657a69732e ip 00007ff8292d5a68 sp 00007ffe03c0f1b0 error 4 in libhsa-runtime64.so.1[d4a68,...]
10:38:14 kernel: traps: strata-host[196782] general protection fault ip:729ea84d5a65 ... in libhsa-runtime64.so.1[d4a65,...]
10:38:16 control: the engine stopped unexpectedly (exit code -11). ... a hang it caught.
10:38:57 control: the engine stopped unexpectedly (exit code -6). ... a hang it caught.
```

All faults are at the same two offsets inside `libhsa-runtime64.so.1` (`d4a65` / `d4a68`). The bundled ROCm runtime did not change between the two engines: `libhsa-runtime64.so.1` and `libamdhip64.so.7` have identical md5 sums in `components/engine/0.1.57-linux-x86_64-hip-rx9070/` and `0.1.58-linux-x86_64-hip-rx9070/`; only the `strata` binary differs (build ids `0bdcef4682c5+rocm` -> `7be42963e6be+rocm`).

Control restarted the engine after each crash in about 40 s, so the server stayed usable until 10:38. The 10:38:14 crash happened while slot 1 was reading a 91,102-token prompt (`/status` kept showing `"phase": "reading the prompt", "prompt_tokens": 91102, "generated": 0`), and the second crash 40 s later, during the restart, was followed by kernel memory corruption:

```
10:38:57 kernel: BUG: Bad page map in process strata-pool-3  pte:8000000778398025
10:38:57 kernel: page: refcount:4 mapcount:-1 mapping:000000007dc5f2da index:0x25cda0 pfn:0x778398
10:38:57 kernel: head: order:2 mapcount:-1 entire_mapcount:0 nr_pages_mapped:8388607 pincount:0
10:38:57 kernel: aops:ext4_da_aops ino:1a30f8 dentry name(?):"Qwen3.8-Flash-Next-GSQ-RCO-abliterated-IQ3_S-00002-of-00002.gg"
10:38:57 kernel: file:Qwen3.8-Flash-Next-GSQ-RCO-abliterated-IQ3_S-00002-of-00002.gguf fault:filemap_fault mmap:0x0 mmap_prepare: ext4_file_mmap_prepare read_folio:ext4_read_folio
10:38:57 kernel: CPU: 4 UID: 1000 PID: 206423 Comm: strata-pool-3 Kdump: loaded Tainted: G S         O        7.0.0-42-generic #42-Ubuntu PREEMPT(lazy)
10:38:57 kernel: Call Trace: print_bad_page_map.cold / zap_present_ptes / zap_pmd_range / entry_SYSCALL_64_after_hwframe
10:38:57 kernel: BUG: Bad page map in process strata-pool-3  pte:800c00077819a025   (two more, same file, then)
10:38:57 kernel: BUG: Bad page state in process strata-pool-3  pfn:7781cd ... page dumped because: non-NULL mapping
10:39:02 kernel: BUG: Bad rss-counter state mm:00000000181fa605 type:MM_FILEPAGES val:1 Comm:khugepaged Pid:124
10:44:01 kernel: INFO: task (sh):208117 blocked for more than 122 seconds.
10:46:03 kernel: INFO: task wpa_supplicant:1900 blocked for more than 122 seconds.
10:48:06 kernel: INFO: task systemd:1 blocked for more than 122 seconds.
10:48:06 kernel: INFO: task NetworkManager:1896 blocked for more than 122 seconds.
13:04:02 kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: Tasks blocked on level-0 rcu_node (CPUs 0-15): P208074 (detected by 15, t=8700242 jiffies, ...)
```

The pages are the second GGUF file (the PLE table, read with `--ple-io ram`; the engine prints at start that the 26.8 GiB table is not locked because RLIMIT_MEMLOCK is the default, so its pages are reclaimable). From 10:39 on, the crashed engine could not be reaped: control logged `the engine had stopped (exit code None); starting it again (a minute or two) ...` every 30 s until the power was cut, `/health` answered 200 with `"loaded": false`, `/v1/messages` answered 503 `Strata is still releasing GPU/RAM; retry unloading after it exits`, and new processes could not start (sshd accepted TCP connections but never sent its banner). Only a power cycle (13:04) recovered the machine. I report this part for completeness: the kernel side is not Strata's code, but it was the crash of 0.1.58 that triggered it, and 0.1.57 never crashed here.

Traffic at the time (from `/metrics`, two Anthropic-API clients, `--parallel 2`): prompts of 35K to 92K tokens with conversation-cache reuse (83K to 88K reused), outputs up to 4,500 tokens, decode 27 to 40 tok/s when healthy. The last requests before the fatal crash:

```
10:34:09 dur=213.8 finish=stop  prompt=35647 reused=34571 out=4501 tok_s=26.9
10:34:36 dur=184.9 finish=stop  prompt=87734 reused=86315 out=2180 tok_s=39.9
10:36:03 dur=132.8 finish=error prompt=84541 reused=83284 out=380
10:36:59 dur=61.1  finish=stop  prompt=38170 reused=36471 out=114  tok_s=9.0
10:37:16 dur=36.3  finish=stop  prompt=84700 reused=83289 out=45   tok_s=4.4
10:37:43 dur=26.8  finish=stop  prompt=40205 reused=35642 out=374  tok_s=34.6
10:37:58 dur=15.3  finish=stop  prompt=90945 reused=87734 out=85   tok_s=37.8
10:38:10 dur=5.9   finish=error prompt=40695 reused=None  out=0
10:38:11 dur=46.3  finish=error prompt=91102 reused=None  out=0
```

### Smallest steps or command that reproduce it
No on-demand reproduction. It happened 9 times in 5 hours of ordinary serving with the settings below (two concurrent clients, 35K to 92K-token prompts, conversation cache on). With engine 0.1.57 and the identical run config, traffic and machine, zero crashes before (23 h) and after the switch back.

Workaround that worked: `strata components use 0.1.57 --family qwen --model-config <run config>` (status "switched"), then point `vision.exe` / `vision.cpu_exe` in the run config at the 0.1.57 folder and restart `strata serve`.

### Reproduction status
Reproduced locally

### Strata version
Strata 0.2.12 · control 0.1.58 · engine 0.1.58-linux-x86_64-hip-rx9070 (BUILD.json source `f-amd-vision 7be42963e6be`). Clean on engine 0.1.57 (`f-amd-vision 0bdcef4682c5`).

### Graphics card, memory and system
Radeon AI PRO R9700 32 GB (gfx1201) · Ryzen 7 9800X3D · 128 GB RAM (2x64 GB) · Ubuntu, kernel 7.0.0-42-generic · ASUS ProArt X870E-CREATOR WIFI (BIOS 2503) · 8 GB swap file, `vm.overcommit_memory=0`

### Model, size and settings
Qwen3.8-Flash-Next GSQ-RCO abliterated IQ3_S (pack `sc117_iq3_s`, two GGUF files; second file = PLE table) · `--max-context 262144` · `--kv int8` · `--mtp` · `--spec 4 --spec-min-p 0.5` · `--expert-cache auto` · `--prefill auto` · `--ple-io ram` · `--kv-resident 32768` · `--conversation-cache-mib 20480 --conversation-cache-slots 6` · `--vision` · `--vram-reserve-mib 700` · `reasoning_budget_tokens 4096` in the run config · `serve --parallel 2` · env `STRATA_RESIDENT_PIN=0` · RLIMIT_MEMLOCK at the default (the engine logs "the PLE table (26.8 GiB) was not locked in RAM")

### Logs
Engine log right before the fatal crash (the engine had just restarted at 10:38:16 and was starting up again at 10:39:00; the session line reads `strata generate: session is up (engine 0.1.58)`):

```
strata serve: conversation cache: restored 87734 tokens (live) in 83.5 ms; parked=5 bytes=9411165984
strata serve: conversation cache: slot 1 not parked before its overwrite (its 40578 tokens have no per-slot draft K/V without --batch-mtp)
strata batch: slot 1 takes 90945 tokens (copied in 177.7 ms)
strata serve: prompt 90945 tokens = 87734 reused + 3211 read in 1531 ms (2096.7 tok/s), 1 generated in 19 ms (53.3 tok/s), drafts accepted 0 of 0, 6 checkpoints
strata serve: decode expert cache hit rate: 93.1% (447 hits / 480 lookups)
strata serve: KV streaming: 56.71% of 30816 block reads hit VRAM, 53.7 MiB read from RAM
strata serve: conversation cache: parked 90945 tokens in 52.8 ms; parked=5 bytes=9555374900 evictions=35 snapshot_bytes=2335177888 reused_kv_bytes=1339141248
strata serve: conversation cache: restored 40287 tokens (live) in 41.0 ms; parked=5 bytes=10170497360
[strata] 2026-11-08 10:38:16 engine started: ~/.local/share/strata/components/engine/0.1.58-linux-x86_64-hip-rx9070/strata --serve --pack ...
```

Start-up lines of 0.1.58 on this card (for the kernel selection):

```
strata: gfx12 (RDNA4): exact extra switches on by default (STRATA_GFX12_DEFAULTS=0 turns them off; a switch you set is kept): HEAD_MIX_MULTI
strata: gfx12 (RDNA4): the matrix-core QSA block scorer is on once a prompt is past 28,672 tokens
strata: gfx12 (RDNA4): 12 exact speed switches on by default: GDN_HEAD GDN_PP GDN_CONVL2 GDN_NOY CVEC_FUSE ATTN_LANECELL ATTN_MERGE_V2 TSUM EXPDEC ...
strata generate: PCIe probe: 48.9 GB/s host->device -> pcie_frac 0.15 (default 0.15)
strata: the PLE table (26.8 GiB) was not locked in RAM: this program may lock only 0.0 GiB (RLIMIT_MEMLOCK). It still works, with the table's pages faulted in but reclaimable
strata generate: expert cache 11324 slots, 21.49 GiB of VRAM
strata serve: prompt chunk auto: 32768 tokens, a 96-slot ring
strata mtp: draft head over 106299 tokens (212.9 MiB)
strata serve: 842 MiB of VRAM free with everything loaded
strata: prompt experts on the fused int8 kernels (STRATA_PF_FUSED=1, #136)
```

I can run `strata doctor`, collect a core dump, or try 0.1.58 again with a specific switch (for example `STRATA_GDN_CHUNKED=0`) if that helps narrow it down.

関連リンク

インストール・モデル・リリースへのサイト内リンク。