#5672 [Engine] gfx1100 (RX 7900 XTX) --adapt-async 1 results in timeout followed by a display-driver reset during long generations
closed · @wburzyns · 4 Kommentare · Auf GitHub
BenchmarksSetup & installServer & APIAMD / HIPNVIDIA / CUDAModels & quantsWindows
Beschreibung
Summary
With --adapt-async 1, long generations on an RX 7900 XTX (gfx1100, Windows, HIP) end with the engine dying on a verify timeout, and Windows resets the display driver (Event 4101, then the AMD Bug Report Tool pops up). It happens every 1–3 long answers (anywhere from 5 to ~60 minutes of generating).
Removing --adapt-async 1 makes it go away and changes nothing else: the same engine and config run clean. A step-by-step bisect isolates the option (details below). It still happens on 0.2.13 (engine 0.1.59).
Without the option, decode speed on this setup is the same (about 72–78 tok/s with MTP on long answers either way), so for me the workaround costs nothing. I'm reporting it because the failure takes the whole display driver down with it.
Error
[strata] the engine stopped unexpectedly (exit code None). Unrecoverable native verification failure: verify: timed out at layer 13; its GPU waits were released but the GPU did not finish within 5 s (#267) The next request starts the engine again.
At the same moment Windows logs System Event 4101 (Display driver amdkmdag stopped responding and has successfully recovered).
System
GPU: Sapphire RX 7900 XTX 24 GB (Navi 31, gfx1100), PCIe 4.0 x16, Resizable BAR disabled
Driver: Adrenalin 26.3.1 (32.0.23033.1002)
GPU tuning: undervolt 1050 mV; VRAM clock 2650 MHz (default 2500), set about a year ago. The failure also happens at 1150 mV. Not yet tested at stock VRAM clock; I can do that if useful.
OS: Windows 10 1809 (build 17763)
RAM: 128 GB; CPU without AVX-512 (7 pool workers + host thread)
Model: Qwen3.8-Flash-Next GSQ-RCO IQ3_S (ISTA-DASLab), MTP drafter (--spec 4 --spec-min-p 0.7); also seen with DFlash2
Engines where it happened: 0.1.56, 0.1.57, 0.1.58, 0.1.59 (app 0.2.13)
Bisect
Same long prompt every run (a "write a full Java limit order book with tests" prompt that produces 60k–200k tokens), driven by a script that repeats it and checks the Windows event log for driver resets after each run.
# Server Engine Config Result
1 older copy (engine 0.1.44.1 era) 0.1.44.1 baseline config (below) 14 / 14 clean
2 older copy 0.1.56 my current config: baseline + --adapt-async 1, STRATA_ROUTE_TAIL_SKIP=7, STRATA_PREFILL_CPU_SHARE=auto, STRATA_PREFILL_CPU_SHARE_MAX=4096, STRATA_SELECT_WMMA=1, STRATA_HIPBLASLT_TABLE=on crash in run 1 (19 min)
3 older copy 0.1.56 baseline config, only the engine swapped 14 / 14 without crash (12 full answers, 2 stopped early by the repeated-token stop)
4 older copy 0.1.56 #3 + --adapt-async 1 + ROUTE_TAIL_SKIP=7 + PREFILL_CPU_SHARE=auto + PREFILL_CPU_SHARE_MAX=4096 crash in run 1 (5 min)
5 older copy 0.1.56 #3 + --adapt-async 1 only crash in run 3 (after 26 + 29 + 6 min)
6 app 0.2.12 0.1.58 my full current config minus --adapt-async 1 10 / 10 clean
7 app 0.2.13 0.1.59 my full current config (with --adapt-async 1) crash
8 app 0.2.13 0.1.59 my full current config minus --adapt-async 1 clean
Rows 3 and 5 differ only by --adapt-async 1. Row 2 also rules out the server version: the failing config crashes under the older server too.
Baseline config (row 3)
--pack ...\packs\iq3_s --native ...IQ3_S-00001-of-00002.gguf --ple-gguf ...IQ3_S-00002-of-00002.gguf
--expert-profile ...\expert-profile.bin --expert-cache auto --prefill auto:32768
--spec 4 --spec-min-p 0.7 --mtp ...\mtp\rt --max-context 204800 --kv int8 --kv-resident 32768
--pool-workers 7 --vram-reserve-mib 768
env: STRATA_HIPBLASLT_TUNING=<gfx1100-hipblaslt-100500.txt>, STRATA_WDDM_BUDGET=0, STRATA_PF_FUSED=1, STRATA_PF_GEMM=1, STRATA_PF_SWITCH_MIN_T=4096, STRATA_HIP_WMMA=1, STRATA_PA_FAST=1
Relevant log lines (failing run, 0.1.56, MTP)
strata serve: adaptive tier asynchronous (--adapt-async 1): up to 96 swaps per round, a round every 4 windows (arena: copy-in only)
...
strata serve: asynchronous adaptive tier: 44447 rounds, 2334441 experts swapped in, 27.0 ms per round (start to flip, between windows) (arena, copy-in only, batch 0, wait 1, pick inline 0.12 ms/round)
In the clean runs the cache is static ("PROFILE, ranked by routing frequency, no eviction"). Other observations:
Both clean and failing runs start with about 807 MiB of VRAM free and print the #2065 low-VRAM warning (--vram-reserve-mib 768). So low headroom alone doesn't cause it, but it may matter in combination with the async swaps.
In one failing start the arena pin partly failed (cudaHostRegister of the whole arena FAILED (out of memory); 26 slices pinned, #3431 alias warnings). Other failing starts pinned the arena fine (cudaHostRegister PORTABLE ok; large pages), so that isn't required for the crash.
The part I can't explain
--adapt-async 1 was on from 29 Oct to 5 Nov (engines 0.1.52–0.1.57, MTP): about 3.7 hours of generating, 23 answers, with no crash. Since 6–7 Nov it crashes every 1–3 long answers. At the current rate that clean stretch would be under a 1% chance. No Windows or driver updates were installed in that period (only Defender signatures), and the hardware and tuning didn't change.
Did anything change in the async adaptive tier, or in what it depends on (expert profile, routing statistics, arena layout), in builds or data from around 6 Nov?
Ruled out
VRAM: memtest_vulkan and OCCT VRAM test clean
Model files: SHA-256 of both GGUF shards match the published ones; packs rebuilt, identical
No WHEA errors; temperatures normal (VRAM 88–90 °C under Strata)
Voltage: fails at 1150 mV too
Drafter: fails with MTP and DFlash2
Experimental speed projection: fails with it removed
Server version: fails under the older server too (row 2)
Mehr auf der Site
Links zu Install, Modellen, Releases.