反馈 / #2157

#2157 [Bug]: 2x Arc Pro B70 layer split, every expert in VRAM: DEVICE_LOST at prefill.cpp:4678 reading a prompt (0.1.43 and 0.1.43.1; 0.1.42 passes)

open · @deriknel · 3 评论 · 去 GitHub 看

BenchmarksSetup & installServer & APIMulti-GPUModels & quants

说明

[crash-0143rt-2010-engine.log](https://github.com/user-attachments/files/33351837/crash-0143rt-2010-engine.log)
[crash-0143rt-2010.log](https://github.com/user-attachments/files/33351841/crash-0143rt-2010.log)
[crash-0143rt-ce0-2015-engine.log](https://github.com/user-attachments/files/33351839/crash-0143rt-ce0-2015-engine.log)
[crash-0143rt-ce0-2015.log](https://github.com/user-attachments/files/33351842/crash-0143rt-ce0-2015.log)
[crash-0143rt-oldrank-2028-engine.log](https://github.com/user-attachments/files/33351840/crash-0143rt-oldrank-2028-engine.log)
[crash-0143rt-oldrank-2028.log](https://github.com/user-attachments/files/33351843/crash-0143rt-oldrank-2028.log)
[crash-04331-2050-engine.log](https://github.com/user-attachments/files/33351845/crash-04331-2050-engine.log)
[crash-04331-2050.log](https://github.com/user-attachments/files/33351844/crash-04331-2050.log)
[longctx.py](https://github.com/user-attachments/files/33351838/longctx.py)

### What happened

On two Arc Pro B70s with `--layer-split` and every expert resident, 0.1.43 and 0.1.43.1 lose the device at the start of reading a prompt, after a long prompt has been served:

```
level_zero backend failed with error: 20 (UR_RESULT_ERROR_DEVICE_LOST)Exception caught at file:/opt/strata/sycl/src/prefill/prefill.cpp, line:4678
[strata] reading the prompt: 14,322 tokens, 10 s so far
...
strata serve: no progress for 60 s during a request (reading the prompt (batched): waiting for the GPU (attention, router) at layer 0 of the prompt chunk from token 0) - stopping the engine so the server starts it again (issue #29)
```

The server restarts the engine and later requests work. 0.1.42 on the same machine, with the same model, flags and GPU runtime, does not crash.

**Reproduce** (one server, no restart between steps), with the attached `longctx.py` run inside the container (it reads the image's own sources as filler and calls `localhost:8080`):

1. `python3 longctx.py 600000`: a ~147K-token prompt, one fact planted at 30% depth, asked back with thinking off, temperature 0, `max_tokens` 300. A random first line makes every prompt a cold read. This always succeeds.
2. `for i in $(seq 0 19); do python3 longctx.py $((52000 + i*1000)); done`: 20 prompts of 13.5K to 14.5K tokens.

| Engine | Big prompt | 13K prompts | DEVICE_LOST | Log |
|---|---|---|---|---|
| 0.1.42 (61b3fb5d) | found | 20 of 20 found | 0 | |
| 0.1.43 (7695859a) | found | crashed on the 1st | 4 in the run | `crash-0143rt-2010` |
| 0.1.43, `STRATA_COPY_ENGINE=0` | found | crashed on the 1st | 1 | `crash-0143rt-ce0-2015` |
| 0.1.43, `--expert-profile expert-profile.bin` | found | crashed on the 6th | 1 | `crash-0143rt-oldrank-2028` |
| 0.1.43.1 (a0116b27) | found | crashed on the 4th | 1 | `crash-04331-2050` |

All on Level Zero compute runtime 26.35.39758.10 with IGC 2.41.5. So it is not the copy engine or the new expert ranking. Decode speed is the same on every build (about 72 / 96 / 72 tok/s on prose / code / long), so only the prompt path is affected. The conversation cache was on in every run, and the crashes came with 4, 6 and 8 conversations parked, so a full cache is not the trigger.

Launch command (0.1.43.1 run):

```
strata \
  --serve \
  --native <Swift-1.5-Flash-Next IQ3_S GGUF> \
  --ple-gguf <same GGUF> \
  --pack <pack> \
  --mtp <mtp> \
  --expert-profile /opt/strata/data/expert-profile-sycl.bin \
  --expert-cache auto \
  --stream-experts \
  --prefill 4096 \
  --spec 4 \
  --spec-min-p 0.5 \
  --max-context 524288 \
  --kv int8 \
  --kv-resident 32768 \
  --vram-reserve-mib 1700 \
  --no-prefill-borrow \
  --rope-scaling yarn \
  --rope-scale 2 \
  --conversation-cache-mib 12288 \
  --conversation-cache-slots 8 \
  --layer-split 26
```

Environment: `STRATA_STAGE_TRIM=1 STRATA_VERIFY_NO_HOST=1 STRATA_VERIFY_DEVICE_PLAN=1 STRATA_MIRROR_MIB=0 STRATA_STAGER_THREADS=12` (0.1.42 also has `STRATA_SYCL_GEMM_NO_STAGE=1`). The start-up log reports `layer split: 100% of the experts resident`.

We run the engine in a Docker image built from each release tag (oneAPI 2026.1.1, AOT for bmg-g31), not through setup. We can run any build or setting you want tried; the reproducer takes about 15 minutes.

### Strata version, GPU, OS

0.1.43.1 (a0116b27) and 0.1.43 (7695859a) · 2x Arc Pro B70 32 GB, xe driver, GuC 70.72.1 · CachyOS, kernel 7.2.9, i9-14900KF, 32 GB RAM · Level Zero compute runtime 26.35.39758.10, IGC 2.41.5

本站相关内容

相关页面的快捷入口。