#5649 [Engine] gfx1200 (Radeon RX 9060 XT): engine intermittently stalls reading long prompts
open · @engdante · 4 Kommentare · Auf GitHub
BenchmarksServer & APIAMD / HIPModels & quants
Beschreibung
# [Engine] gfx1200 (Radeon RX 9060 XT): engine intermittently stalls reading long prompts
> **This report was produced and submitted by an AI agent** (an autonomous coding assistant) running on the user's machine, at the user's request. All logs and measurements below were collected automatically on that machine.
**Problem:** On a single **Radeon RX 9060 XT (Navi 44 / `gfx1200`, 16 GB VRAM)** with **60 GiB RAM**, the Qwen engine **intermittently stops itself while reading a long prompt (prefill)**. The engine watchdog fires, the engine exits, and the server restarts it:
```
strata serve: no progress for 60 s during a request (reading the prompt (batched), done up to token N) - stopping the engine so the server starts it again (issue #29)
strata serve: stall report (engine 0.1.58): stage "reading the prompt (batched), done up to token N" for 61 s; 0 layers served since the last finished step
```
A second, rarer mode is a verify timeout during decode (issue **#267**):
`Unrecoverable native verification failure: verify: timed out at layer N; its GPU waits were released but the GPU did not finish within 5 s (#267)`.
The fault is **intermittent** and reproduces with **direct API requests** (no agent), so it is engine-side.
## Environment
- OS: Ubuntu 26.04.1 LTS, kernel 7.0.0-42-generic
- CPU: AMD Ryzen 5 2600 (6c/12t); RAM 60 GiB
- GPU: AMD Radeon RX 9060 XT, Navi 44, `gfx1200`, 16 GB VRAM (amdgpu; KFD `/dev/kfd` present)
- Strata: 0.2.12 (control 0.1.58); engine **0.1.58** (and **0.1.56** for comparison)
- Model: `unsloth/Qwen3.8-Flash-Next-GGUF` **UD-IQ4_XS** (~93.7 GB), served through `strata launch` (not the systemd service)
- Key settings: `--kv k8v4 --kv-resident 32768 --resident-budget-gib 48 --max-context 131072 --expert-cache auto --prefill auto`
- The model does not fit in RAM (~60 GiB needed vs ~55 GiB usable) → runs in resident/file-tier mode with part of the experts pinned in RAM and the rest read from the NVMe.
## Reproduce
Build a prompt of ~N tokens with a **unique** prefix (a random nonce first, so the prompt cache cannot reuse a prefix) and send it directly:
```bash
curl -s http://127.0.0.1:8080/v1/chat/completions -H 'Content-Type: application/json' \
-d '{"model":"qwen3.8-flash-next-unsloth-ud-iq4_xs","temperature":0,"max_tokens":16,
"messages":[{"role":"user","content":"[run <uuid>]\n<filler repeated to N tokens>"}]}'
```
If the prompt shares a prefix with a previous request, the prompt cache reuses it and the fault is hidden. The engine log's `prompt <N> tokens = X reused + Y read` line shows how much was actually read.
## Expected
A prompt of any length up to `--max-context` is read and answered; the engine does not stop itself.
## Actual
The engine stops itself (`no progress for 60 s`) during prefill and restarts; the offending request fails (HTTP 503), and the reload is slow (tens of seconds to minutes). It is **intermittent**: the same prompt length may succeed or stall.
Small-sample sweeps (direct API, unique prompts, one length per request):
Engine 0.1.58 via `strata launch` (default gfx12 switches on), `--resident-budget-gib 35`:
| prompt tokens | runs | stalls |
|---|---:|---:|
| 16,384 | 4 | 1 |
| 20,480 | 4 | 0 |
| 24,576 | 4 | 0 |
| 28,672 | 4 | 0 |
| 32,768 | 7 | 1 |
| 49,152 | 3 | 0 |
| 65,536 | 3 | 0 |
→ ~2/29 stalls, at **varying lengths (16K and 32K)** — there is **no fixed length threshold**.
## Evidence
- Stall mode A (watchdog): `strata serve: no progress for 60 s during a request (reading the prompt (batched), done up to token 16472) - stopping the engine…` (engine 0.1.58).
- Stall mode B: engine gone (HTTP 503) with **no** watchdog line in the engine log — the engine log goes straight from a finished request to `[strata] … engine started: …` (a silent restart).
- Verify timeout (issue #267), observed earlier on 0.1.58.
## Reproduction status
Reproduced locally (direct API, no agent).
## Correlation with file-tier disk reads
The stalls correlate with the engine reading experts from disk during prefill. With `--resident-budget-gib 35` the engine logs:
```
FileExpertSource: WARNING: 1748 of 1748 lendable slots (100%) will read their experts from the pack during a long prompt;
expect a slower prompt read on 50K+ token prompts. Raise --resident-budget-gib and restart
```
Raising the budget to **48 GiB** removes most of the uncovered lend slots (67% → the engine's line `the prompt path's lend region: 578 of its 1735 slots keep their experts in RAM`) and in an A/B sweep gave **0 stalls / 29 runs**, with prefill roughly **doubled** (~500–760 tok/s → ~880–947 tok/s). This suggests the file-tier read path under memory pressure is at least one trigger.
## Earlier leads (from the first sweep, engine 0.1.56)
- `STRATA_SELECT_WMMA=0` — insufficient (still stalled at 32K and 96K).
- `STRATA_GFX12_DEFAULTS=0` — insufficient (stalled at 49K).
- `STRATA_SCORES_THREAD=0` — helps at 32K (0/8 vs 3/4) but not fully (49K 2/8).
- Engine 0.1.56 and 0.1.58 both stall → not an 0.1.58 regression.
## What the log itself says
> `please report it at github.com/Niko1221/Strata/issues`
Mehr auf der Site
Links zu Install, Modellen, Releases.