#6445 [Engine] Abort on UR_RESULT_ERROR_DEVICE_LOST at the verify window (Intel Arc Pro B70, Linux)

open · @sneakyjoeru · 4 コメント · GitHub で見る

Server & APIModels & quantsLinux

本文

This is agentic report after multiple debug sessions. LLM generated. 

## Environment

- Host: Fedora (kernel 7.3.4-200.fc44), headless, one Intel Arc Pro B70 24 GB (`0000:03:00.0`), level_zero 1.x
- Strata: app 0.2.17, engine **0.1.63** (stock SYCL, `strata update`) — reproduced identically on a locally pinned
  engine build from **0.1.59** (`~/strata-engines-pin`), so it is not a regression in one build
- Launch: `strata serve --pack hh_abl_iq3_xxs --kv int8 --kv-resident 229376 --stream-experts --prefill 4096
  --conversation-cache-mib 12288 --conversation-cache-slots 2 --max-context 262144 --vision` (wrapper
  `strata-engine-0159pin.sh`, `--draft dflash2 --spec 6`)
- ~15 aborts over 2026-11-17..20; 3-4/day under normal use, more under deep context

## Symptom

The engine process aborts (SIGABRT, SYCL terminate) and the serve layer returns HTTP 500 with

```
strata verify: window sync: the queue wait failed on verifier device 0
(queue device: Intel(R) Arc(TM) Pro B70 Graphics): level_zero backend failed with error: 20
(UR_RESULT_ERROR_DEVICE_LOST)
terminate called after throwing an instance of 'sycl::_V1::exception'
  what():  level_zero backend failed with error: 20 (UR_RESULT_ERROR_DEVICE_LOST)
```

The matching driver side (dmesg, same second) shows the xe job that the engine was waiting on being killed:

```
[Fri Nov 20 23:38:54 2026] xe 0000:03:00.0: [drm] Tile0: GT0: Engine reset: engine_class=ccs, logical_mask: 0x1,
                                       guc_id=22, state=0x3
[Fri Nov 20 23:38:54 2026] xe 0000:03:00.0: [drm] Tile0: GT0: Timedout job: seqno=474, lrc_seqno=474, guc_id=22,
                                       flags=0x20 in strata [55338]
[Fri Nov 20 23:38:54 2026] Workqueue: gt-ordered-wq drm_sched_job_timedout [gpu_sched]
[Fri Nov 20 23:38:54 2026]  guc_exec_queue_timedout_job+0x289/0xd70 [xe]
[Fri Nov 20 23:38:54 2026] xe 0000:03:00.0: [drm] Xe device coredump has been created
```

So: a CCS job the engine submitted did not complete inside the xe job timeout (we already run the maximum,
`job_timeout_ms = 10000`), the driver reset the context, level_zero reported DEVICE_LOST, and the engine died instead of
retrying or resetting its own state.

A second, related failure mode seen on the same box: a cold 52k-token prefill makes **no progress at all**
("reading the prompt: 52,556 tokens, 10 s ... 80 s so far", prompt_read frozen) until the client gives up (133 s later);
the engine stays busy for the rest of the request.

## What we tried (all negative, details available)

- `pcie_aspm policy=performance`, `STRATA_SPIN_MAX` 20000 vs 2000000, killing the resident vision context
- `ccs preempt_timeout_us` 640000 -> 10000000 (prefill slowed, still stalls), `job_timeout_ms` already at the 10 s max
- kernels 7.3.3 -> 7.3.4 -> 7.3.5, engines 0.1.59 -> 0.1.62 -> 0.1.63
- `--prefill 4096` (in use) and 2048/8192
- No temperature/power throttling visible (GPU 39-53 C, 5-55 W of a 275 W limit); RAM is not pressure-limited (13-16 GB free)

## Questions

1. Is there a supported way to make the verify window's queue wait recover from a xe context reset instead of
   `std::terminate` (e.g. a retry, or `STRATA_VERIFY_*` knob that avoids the separate verifier queue)?
2. Are there env knobs that change how the verify window submits / waits (we found `STRATA_ADAPT_NOWAIT`,
   `STRATA_COMMIT_SYNC`, `STRATA_DOORBELL_CHECK`/`STRATA_DOORBELL_PLAIN` but no documentation on what they do at the
   verify window)?
3. Is the frozen-prefill case (prompt_read not advancing) the same root cause on Xe, and is a smaller `--prefill`
   or a different engine class expected to help?

## Attachments to add when posting

- `~/strata/strata-flashnext-hh-abl-iq3xxs.log` (contains the 15 aborts)
- `sudo dmesg -T | grep -A20 "Timedout job"`
- `/sys/class/drm/card1/device/devcoredump/data` from the last reset (large)

関連リンク

インストール・モデル・リリースへのサイト内リンク。