Issues / #2194

#2194 SYCL, Arc A770 (xe): decode intermittently never starts - host waits in Verifier::run on a queue that does not drain, RSS grows until OOM

open · @aslater3 · 0 comentários · No GitHub

BenchmarksModels & quantsLinux

Descrição

**Hardware:** Arc A770 16 GB (Alchemist, `xe`), Linux 7.0, oneAPI 2026.1 (JIT build), Ubuntu 26.04.1, compute runtime (NEO, `libze-intel-gpu1`) 26.05.37020.3, Ryzen 7 3800X, 30 GB RAM.
**Build:** `main` at `a0116b27` + #2190 (without it this card gets 0 expert-cache slots and cannot start this config).
**Config:** Coder IQ1_M, `--pack ... --native ... --ple-gguf ... --mtp ... --expert-profile <24-trace profile> --expert-cache auto --resident-experts --max-context 131072 --kv int8 --kv-resident 20480 --vram-reserve-mib 1200 --prefill-borrow --prefill auto --spec 4 --spec-min-p 0.5 --adapt-every 8 --tokens-file <8K or 2K prompt> --greedy --max-new 256 --stats`, env `STRATA_VERIFY_STEPPED=1 STRATA_RESIDENT_HEADROOM_GIB=5..7 UR_L0_ENABLE_RELAXED_ALLOCATION_LIMITS=1`, `memlock` unlimited.

## What happens
Intermittently, decode never makes progress. The prompt is read normally, then the log stops after

```
strata verify: window up to 6 tokens, 75.0 MiB of device buffers
strata generate: window and draft graphs captured in 1109 ms
strata verify: pipelined pool helper on CPU 8 (host thread CPU 0)
```

(once it stopped later, after `position 8158, 171 tokens, 64 rounds`). The host thread spins at ~90% CPU, the process RSS keeps growing (~17-18 GB anon at start of decode, 19.4-20.4 GB at the end) until the kernel OOM killer kills it after 5-8 minutes. Nothing from `xe` in the kernel log (no GT reset, no job timeout).

Stack of the main thread in the hang (gdb, two samples a minute apart, identical):
```
#0  sched_yield ()
#1-2 libze_intel_gpu.so.1
#3  ur_queue_handle_t_::synchronize()
#4  ur::level_zero::urQueueFinish
#6  sycl::_V1::detail::queue_impl::wait
#7  strata::core::Verifier::run(...)
#8  strata::core::Verifier::run(...)
#9  main ()
```
The 7 `ExpertPool::worker` threads are parked in `pthread_cond_wait`; the 32 stager threads and the 64 `DirectFile` readers are idle. So the host waits for a queue that never drains: something in the window never completes.

## How often
Same machine, same session, isolated runs (each in its own systemd unit):
- `main` + #2190, default settings: 6 of 7 runs hung (8K and 2K prompts). Three more runs with `STRATA_DOORBELL_CHECK=1`, `STRATA_LEND_MIRROR=0` and `STRATA_VERIFY_STEPPED=0` (one each) finished.
- With the GR_PICK fix (#2193), decode is ~4x faster and 1 of ~15 runs hung (a 2K prompt).
- Our fork build from 2026-10-09 (the stepped window before it was ported, plus our other A770 kernel changes) ran this config today once without hanging; our October notes for this card record no such hang.

## Not tested
`STRATA_VERIFY_STEPPED=0` repeatedly (one run finished, 14.1 tok/s), i915, other Alchemist cards.

Happy to run anything that would narrow it down (a debug build, `ZE_DEBUG`, a specific env switch).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

No site

Links install, modelos, releases.