#5992 [Engine] a GPU wait that runs out is an error for the window (xe spin bound follows the host)
closed · draft · @popisec · 0 comentários · No GitHub
Multi-GPUAMD / HIPNVIDIA / CUDAModels & quantsSecurityDocumentationWindowsLinux
Descrição
## Problem and result On xe, a window the host plans (CPU-served experts, no device plan) makes the GPU wait for the host at every layer, and that wait is bounded to 20,000 reads (tens of ms). When the host was late the wait fell through silently, the layer ran on the previous layer's plan, and the request ended as `!!!!!` (token id 0) with a poisoned state. After this change a wait that runs out is an error for the window, and the xe bound is 1,000,000 reads whenever a stage may ask the host. Measured on 2x Intel Arc Pro B70 (Linux, xe, Qwen3.8 Flash-Next UD-IQ3_XXS, `--spec 4 --mtp`, 4 parallel requests) with a cut `--expert-cache` so the CPU serves part of the experts: 10 of 10 requests bad with the 20,000 bound, 0 of 10 with `STRATA_SPIN_MAX=2000000`. With the change and no environment variable, `--layer-split auto`, 65536 context, 20 document extractions 4 at a time: 0 expiries, 0 engine restarts. The raw logs of the 10/10 run are not attached; I can add them if useful. ## Related discussion Same symptom as #606 (closed), a different cause on SYCL/xe; also touches the #1473 area (Battlemage without sysfs is left unchanged). The change moves two defaults (a run-out wait becomes an error; the xe bound can be 1,000,000 instead of 20,000), so I am asking for agreement on that scope here and keeping this as a draft until then. ## Approach `wait_flag_ge_kernel`, `wait_flag_ge_or_kernel` and `doorbell_wait_kernel` re-read their flag after the bound and record the site in a per-device host word (`spin_mark_expired`); `Verifier::spin_ok` and `session_loop` take it after the window and return an error. `spin_max(queue)` is no longer frozen at the first launch of a device, and the verifier raises the bound whenever the host can be asked, including a stage that starts 100% resident but can fall back to the doorbell graph (`refresh_ar`). `STRATA_SPIN_EXPIRY=warn` only reports. `docs/INTEL.md` documents both variables. ## Validation **Passed:** * SYCL build (AOT `bmg-g31`, docker), revisions `4bf6574` and `038f667` of the author's branch (the first commit of this PR has the tree of `038f667`): 0 errors. * The second commit (the bound choice as the pure function `spin_bound_for()`, the `fixups.py` rule, `spin_bound_test`): compiles with icpx 2026.2; `spin_bound_test` 15/15, also with `-DSTRATA_SYCL_SPIN_MAX`; `spin_bound_for()` equals the previous logic on all 128 input combinations; the `fixups.py` 7c'' rule reproduces the committed code on the files before the expiry change, is idempotent and leaves the current files unchanged. (Compiled and unit-tested; not re-run on the GPU after this refactor.) * 2x Arc Pro B70, `--max-context 65536 --layer-split 26`, `STRATA_VERIFY_DEVICE_PLAN=1 STRATA_VERIFY_NO_HOST=1`, no `STRATA_SPIN_MAX`, 20 document extractions 4 at a time: 20/20 valid, 0 expiries, 0 restarts. * Same machine, three full runs at `--max-context 262144` (needle up to 249,919 tokens, 4 parallel, 20 extractions each): 60/60, 0 loops, 0 errors. **Failed:** * Before the last three of the author's commits, at 65536 with the device plan on, one request ended with the run-out error at 5 tokens and the engine restarted (the bound stayed at 20,000 there); fixed by the final bound logic in this commit. * After an expiry the engine restarts and the concurrent requests are lost (the existing contract for a failed window). `UR_RESULT_ERROR_INVALID_MEM_OBJECT` was seen once after a discard; not investigated. **Not run:** * Windows / OpenCL (Battlemage without sysfs: still 20,000, and its waits are expected to expire, #1473), i915 / A-series, CUDA and HIP: untested. A run-out wait there is now an error too. * A dead or stopped host: how long a window takes to fail with the bound at 1,000,000 (it is per wait, and a graph has several) was not measured. * The full-model runs above were made before the second commit (a refactor that returns the same bound); they were not repeated on the GPU with it. ## Limits After an expiry the window is not retried. The expiry word is one per device; the host's read-and-clear is atomic, the device write is not. Windows and non-xe behaviour are unverified, as above.
No site
Links install, modelos, releases.