Issues / #2194
#2194 SYCL, Arc A770 (xe): decode intermittently never starts - host waits in Verifier::run on a queue that does not drain, RSS grows until OOM
open · @aslater3 · 0 Kommentare · Auf GitHub
BenchmarksModels & quantsLinux
Beschreibung
**Hardware:** Arc A770 16 GB (Alchemist, `xe`), Linux 7.0, oneAPI 2026.1 (JIT build), Ubuntu 26.04.1, compute runtime (NEO, `libze-intel-gpu1`) 26.05.37020.3, Ryzen 7 3800X, 30 GB RAM. **Build:** `main` at `a0116b27` + #2190 (without it this card gets 0 expert-cache slots and cannot start this config). **Config:** Coder IQ1_M, `--pack ... --native ... --ple-gguf ... --mtp ... --expert-profile <24-trace profile> --expert-cache auto --resident-experts --max-context 131072 --kv int8 --kv-resident 20480 --vram-reserve-mib 1200 --prefill-borrow --prefill auto --spec 4 --spec-min-p 0.5 --adapt-every 8 --tokens-file <8K or 2K prompt> --greedy --max-new 256 --stats`, env `STRATA_VERIFY_STEPPED=1 STRATA_RESIDENT_HEADROOM_GIB=5..7 UR_L0_ENABLE_RELAXED_ALLOCATION_LIMITS=1`, `memlock` unlimited. ## What happens Intermittently, decode never makes progress. The prompt is read normally, then the log stops after ``` strata verify: window up to 6 tokens, 75.0 MiB of device buffers strata generate: window and draft graphs captured in 1109 ms strata verify: pipelined pool helper on CPU 8 (host thread CPU 0) ``` (once it stopped later, after `position 8158, 171 tokens, 64 rounds`). The host thread spins at ~90% CPU, the process RSS keeps growing (~17-18 GB anon at start of decode, 19.4-20.4 GB at the end) until the kernel OOM killer kills it after 5-8 minutes. Nothing from `xe` in the kernel log (no GT reset, no job timeout). Stack of the main thread in the hang (gdb, two samples a minute apart, identical): ``` #0 sched_yield () #1-2 libze_intel_gpu.so.1 #3 ur_queue_handle_t_::synchronize() #4 ur::level_zero::urQueueFinish #6 sycl::_V1::detail::queue_impl::wait #7 strata::core::Verifier::run(...) #8 strata::core::Verifier::run(...) #9 main () ``` The 7 `ExpertPool::worker` threads are parked in `pthread_cond_wait`; the 32 stager threads and the 64 `DirectFile` readers are idle. So the host waits for a queue that never drains: something in the window never completes. ## How often Same machine, same session, isolated runs (each in its own systemd unit): - `main` + #2190, default settings: 6 of 7 runs hung (8K and 2K prompts). Three more runs with `STRATA_DOORBELL_CHECK=1`, `STRATA_LEND_MIRROR=0` and `STRATA_VERIFY_STEPPED=0` (one each) finished. - With the GR_PICK fix (#2193), decode is ~4x faster and 1 of ~15 runs hung (a 2K prompt). - Our fork build from 2026-10-09 (the stepped window before it was ported, plus our other A770 kernel changes) ran this config today once without hanging; our October notes for this card record no such hang. ## Not tested `STRATA_VERIFY_STEPPED=0` repeatedly (one run finished, 14.1 tok/s), i915, other Alchemist cards. Happy to run anything that would narrow it down (a debug build, `ZE_DEBUG`, a specific env switch). 🤖 Generated with [Claude Code](https://claude.com/claude-code)
Mehr auf der Site
Links zu Install, Modellen, Releases.