#6392 [Engine] Linux HIP RX 7900 XTX (32 GB RAM, IQ2_XS low-RAM): engine hangs within 4-10 min of sustained serving, verify timeouts, also on 0.1.63

open · @zverguilov · 3 comentários · No GitHub

BenchmarksSetup & installServer & APIAMD / HIPModels & quantsLinux

Descrição

## Environment

| | |
|---|---|
| Strata | Terminal 0.1.62 / engine 0.1.62 `linux-x86_64-hip-rx7900`; re-tested on 0.2.17 / engine 0.1.63 (same variant) |
| Model | Qwen3.8-Flash-Next **IQ2_XS** (native pack + PLE shard), installed with `--family qwen --model IQ2_XS --context 163840` |
| GPU | AMD Radeon RX 7900 XTX 24 GB (gfx1100), single card |
| CPU | Ryzen 7 5700X3D (8C/16T, AVX2) |
| RAM | 32 GB (31.3 GiB visible), 16 GB swap |
| OS | Bazzite 45 (Fedora 45 Silverblue), kernel 7.2.15, in-kernel amdgpu |
| Mode | setup chose the low-RAM / resident mode (GPU expert cache ~12.2k experts, 15-22 GiB of experts in RAM), `STRATA_RESIDENT_PIN=0` (setup's default for ROCm) |
| Serve | `serve/server.py --engine strata --config <run config> --host 0.0.0.0 --port 8080 --offline`, started by a systemd user unit |

Engine args (from the run config): `--expert-profile ... --expert-cache auto --prefill auto --spec 4 --spec-min-p 0.5 --mtp <mtp> --max-context 163840 --kv int8 --resident-experts`

## Workload

An OpenAI-compatible client sending RAG questions: 2 concurrent workers, prompts of 2-7k tokens (a ~2.1k-token shared system prompt + retrieved excerpts), thinking on with `thinking_budget_tokens` 2000, answers 200-2,600 tokens. Short benchmarks (single 4k-60k prompts, 117k multi-turn) mostly pass; the hang comes under this sustained mixed load.

Speed when it runs is great (parallel 1: ~1,000 tok/s prompt at 60k, 72-83 tok/s decode), so I would really like to keep it.

## What happens

Every run of this workload stopped within **4-10 minutes** (7 runs over ~10 hours, 0.1.62 and 0.1.63). Three forms:

1. `verify: timed out at layer 46; its GPU waits were released but the GPU did not finish within 5 s (#267)` -> engine exits, server restarts it (also `layer 16` with pinned RAM).
2. Stall report: `the prompt's GPU work (GPU 0) had finished, but the engine was still waiting for it` (3 times) / `... was still running` (2 times) -> `no progress for 60 s` watchdog.
3. Engine silent with no stall report: server log `the engine said nothing for 392 s during the request; the server ended the engine` (#481 text) - the last one on **0.1.63**.

Almost every request logs `the graphics driver paused the GPU for 100-3,000 ms during this request while Linux moved memory the GPU uses`.

Representative stall report (0.1.62, swap disabled for the unit, conversation cache on):

```
strata serve: no progress for 60 s during a request (reading the prompt (batched): layer 15 of the prompt chunk from token 2126) - stopping the engine so the server starts it again (issue #29)
strata serve: stall report (engine 0.1.62): stage "reading the prompt (batched): layer 15 of the prompt chunk from token 2126" for 128 s; 0 layers served since the last finished step (0 = stopped, more = slow)
  VRAM free when this request began: 866 MiB
  the prompt's GPU work (GPU 0) had finished, but the engine was still waiting for it
  expert pool: epoch 181994, batch epoch 181994: 24 of 24 jobs claimed, 24 done; 7 of 7 workers parked, 7 sleeping; mode 0
  expert pool threads: w0=sleeping w1=sleeping ... w6=sleeping; host idle for 183401 ms
```

## What I tried (each change, then the same workload again)

| # | Change | Result |
|---|---|---|
| 1 | defaults from setup (`parallel` 2, headroom 6) | strata-host 27 of 31 GB, 10 GB in swap; 127 s stall "parking the previous conversation", 2 watchdog restarts |
| 2 | `parallel` 1, `--conversation-cache-mib 2048`, `STRATA_RESIDENT_HEADROOM_GIB=8` | no more swap storms in a benchmark, but the workload hit form 1 (verify timeout at layer 46) with kernel `kswapd0: page allocation failure` |
| 3 | headroom 12, conversation cache 1024 | ~11 GB MemAvailable, but 5.5 GB of the engine still went to swap -> silent 392 s |
| 4 | + `MemorySwapMax=0` on the unit (no swap for the engine) | form 2, right after the conversation cache's second background spill to the SSD tier |
| 5 | + `--conversation-cache-mib 0` | form 3 (silent 388 s, no stall report) |
| 6 | + `STRATA_RESIDENT_PIN=1` (15.65 GiB page-locked, OK) | form 1 (layer 16) after 4 min; the "Linux moved memory" GPU pauses stayed (up to 3 s) |
| 7 | engine **0.1.63**, settings of #5 | form 3 after ~4 min; GPU pauses still there |

Since the GPU pauses persist with the expert RAM page-locked, my guess is that the moved memory is the mmap'ed pack / model file pages the GPU reads (`experts via mmap (--mmap-experts ...)`, token embedding "in mapped host memory"), and that a KFD queue eviction/restore during a verify window or a prompt chunk loses a completion. Kernel settings on this box: THP `madvise`, THP defrag `madvise`, khugepaged defrag 1, `compaction_proactiveness` 20, no NUMA balancing (single node).

## Questions

- Is there a supported way on HIP/Linux to keep the GPU-mapped file pages from being migrated (e.g. read the pack into RAM instead of mmap, or a setting for this)?
- Would you expect the Coder IQ1_M (more of it in VRAM) to avoid this on a 24 GB card with 32 GB RAM?
- Anything else from the log you would like? I have the full engine log (paths and hostnames removed) and can run a debug build.

No site

Links install, modelos, releases.