#6453 [Engine] SYCL: an integrated Intel GPU is planned as one pool of memory with the system (Arc B390)

open · draft · @criskurtin · 0 Kommentare · Auf GitHub

BenchmarksSetup & installServer & APIAMD / HIPNVIDIA / CUDAModels & quantsSecurityDocumentationWindowsLinux

Beschreibung

## Problem and result

An Intel GPU built into the processor has no memory of its own: its "VRAM" is the system RAM. On main 3142b66b9 an Arc Pro B390 (Panther Lake, `8086:b086`, `xe`, 62 GiB shared) sized the expert cache from Level Zero's free figure, which follows MemFree, and then copied the experts that did not fit into a pinned host mirror in the same RAM. #3729 sizes the cache from MemAvailable less 6 GiB, but a mirror is still built, and the pages the `xe` driver keeps pooled after an unload (`GPUReclaim` in `/proc/meminfo`, 15-23 GiB before these starts) are not counted.

With this PR the cache is sized from MemAvailable + `GPUReclaim` less the headroom, and gets no mirror. Same model and run config for every build, 3 rounds of `terminal/tools/ab_engine.py`, arms alternated, a fresh server per arm:

| build | expert cache at start (3 starts) | pinned mirror | lowest MemAvailable | decode tok/s: code prompt, long prompt (median of 3) | 6,345-token prompt tok/s (median, range) |
|---|---|---|---|---|---|
| main | 8,425-13,400 slots (13.7-21.7 GiB) | 19.2-25.4 GiB | 4.4-8.7 GiB | 19.1, 13.3 | 394.6 (371.1-414.0) |
| main + #3729 | 10,985-15,676 slots (17.8-25.4 GiB) | 16.5-20.9 GiB | 4.5-7.8 GiB | 21.6, 14.7 | 403.5 (292.9-425.1) |
| this PR | 24,576 slots (39.97 GiB) each time | none | 6.2-6.5 GiB | 26.3, 20.4 | 319.2 (289.4-322.8) |

Decode is 38% faster than main on the code prompt and 53% faster on the long prompt's answer, because no expert is read from the mirror.

Prompts are slower on this branch alone: over every run of this config on this machine, a 6,345-token prompt ran at a median of 334.8 tok/s (15 runs) against 394.0 on main (8 runs). My reading of the code: the prompt loan borrows about 1,960 cache slots, and with under 10% of the experts outside the cache the prompt path takes the stream-all walk and reads the borrowed experts back from the file, where main reads most of them from its mirror. Neither the lend mirror (359.0 against 366.9 tok/s) nor routed-only staging (`STRATA_PREFILL_STREAM_ALL=0`: 307.7 against 381.0) recovered it. With #6455 (Xe3 on the Battlemage kernels) on top, the same prompt runs at 582.0 tok/s (571-590).

A model that does not fit is refused right after the cache is sized: with `STRATA_UMA_HEADROOM_GIB=30 STRATA_VERIFY_NO_HOST=1` the server exited 8 s after it started, with "REFUSED: this model does not fit in the memory the integrated GPU shares with the system: 14606 of its experts (about 23.8 GiB) have no room. ...". Main refuses this case only after the fill.

## Related discussion

#6448 (scope for integrated Intel GPUs). Scope is not agreed yet, so this stays a draft.

**Depends on #3729.** The first commit is #3729's, with its author, unchanged except for the paths (`git am --directory=engine`, since `sycl/` moved under `engine/`); the second commit is this PR. Merge #3729 first and this rebases to one commit, or merge this with both. #6455 (Xe3) and #6454 (Monitor) do not depend on this PR and merge cleanly with it in any order.

Overlaps with open PRs: #6369 conflicts only in `terminal/data/settings-register.json`'s line numbers (rerun `tools/settings_register.py` after merging either), and #6319 rewrites the same integrated-GPU paragraph of `docs/INTEL_ARC.md`; both texts can stay.

## Approach

Every change sits behind `device_unified_memory()`, which is true only on a device that #3729 flags as integrated (or with `STRATA_UMA=1`), and only on Linux, like the MemAvailable rule (#409) it replaces.

- **Sizing:** `device_free_bytes()` returns MemAvailable + `GPUReclaim` − `STRATA_UMA_HEADROOM_GIB` (default 6, unchanged), capped at the device's global memory. A kernel without the `GPUReclaim` line, or one that reports 0 (7.0 in #2412), adds nothing. One line at the start prints the figures. The CUDA/HIP rule in `engine/src` is not touched.
- **No second copy in the same RAM:** an auto-sized cache gets no pinned mirror; the experts it cannot hold are read from the file by the host, as main does without a mirror. A cache size you give leaves the pool's rest, less `--vram-reserve-mib`, to the mirror. The lend mirror is off (measured above). KV streaming is not turned on by itself, since it keeps the K/V in pinned RAM and a window of it in the same pool.
- **Refusal before the fill:** with `--stream-experts`, `STRATA_VERIFY_NO_HOST` and an auto cache that cannot hold every expert, the start ends as soon as the cache is sized, naming the experts and GiB that are short.
- **`STRATA_UMA=1` / `=0`** overrides the device's answer. It is in the settings register as opt-in, and `docs/SETTINGS.md` is regenerated.
- The arithmetic is in small inline functions in `engine/sycl/include/strata/core/uma.hpp`; `uma_test` checks it on the CPU (built with `STRATA_SYCL_PARITY=ON`, like `verify_env_test`).

## Validation

**Passed:**
- Arc Pro B390 (Core Ultra X9 388H, 62 GiB LPDDR5X shared, Fedora 44, kernel 7.2.11, compute runtime 26.35.39758), oneAPI DPC++ 2026.1.1, oneMKL 2026.1.0. Packages built with `tools/packaging/build_sycl_linux.sh` from main 3142b66b9, main + #3729 and this branch. Model: the Swift IQ3_XXS files of `huihui-ai/Huihui-Qwen3.8-Flash-Next-abliterated-v2-GGUF` (Swift 1.5's layout, 24,576 experts, 40 GiB). Run config: `--expert-cache auto --stream-experts --prefill auto --prefill-borrow --spec 4 --spec-min-p 0.5 --mtp <rt> --max-context 131072 --kv int8 --vram-reserve-mib 1024`, no `STRATA_*` variables unless named.
- `python terminal/tools/ab_engine.py b390.json main:exe=<main> pr3729:exe=<main+#3729> onepool:exe=<this PR> --rounds 3`: the table above. The greedy answers of the three chat prompts are the same in all three builds. The table was measured before a last correction of comments and docs; a package of the corrected code gave one more round with the same sizing (24,576 slots, no mirror) and the same answers.
- `STRATA_UMA=0` on this build against main, 3 rounds: it sized from Level Zero's free figure and mirrored like main (9,713-13,193 slots, 20.5-25.8 GiB mirrored), at the same speeds (6,345-token prompts 403.1 against 393.3 tok/s).
- Experts outside the cache: with `--expert-cache 16000` (3,115 experts outside), reading them from the file, from a mirror, or from a mirror with `STRATA_VERIFY_NO_HOST=1` all gave correct answers.
- The package build, and a parity build with #6455 and `STRATA_SYCL_PARITY=ON` (299 targets): no errors. `ctest` of that build on the card: `uma_test` passed. 21 other tests failed, the same 21 that main 3142b66b9's own parity build fails on this machine (`ple_parity` needs a Q2_0 model file that is not here, `verify_parity` reads NaN probabilities, and 19 `native_expert_parity_new_*` CPU cases mismatch ggml on this AVX2-only CPU).
- `uv run --python 3.12 --with-requirements terminal/requirements.txt --with-requirements terminal/requirements-dev.txt --with-requirements terminal/requirements-build.txt python tools/check --strict --only cmake-cpu,py-tools,docs` at 1e39caba8: cmake-cpu 137 passed, py-tools 1851 passed, docs 6 passed; 0 failed. `python3 tools/settings_register.py --check`: passed.
- All runs, engine logs and 1-second `/proc/meminfo` logs: https://github.com/xenonhr/Strata/tree/b390-evidence (`memory/`, `uma0/`, `nohost/`, `lend2/`, `stage/`, `stage-xe3/`, `final/`, `engine-tight.log`; its README lists every file).
- Fork CI at the same head, run by hand: [Backend compile checks](https://github.com/xenonhr/Strata/actions/runs/44840605937) (SYCL oneAPI 2026.1, CUDA 12.9 and 13.0, HIP ROCm 7.2, Metal), and every job of [Checks](https://github.com/xenonhr/Strata/actions/runs/44840608849) but the test under Failed.

**Failed:**
- In CI, one test that fails the same way on main 3142b66b9 ([Checks](https://github.com/Niko1221/Strata/actions/runs/44699739037)): `terminal.tools.test_strata_mcp.Install::test_a_start_that_came_in_before_a_cancel_does_not_launch_the_job` on Windows.

**Not run:**
- A discrete Intel card: none here. By code reading the card path is unchanged: `device_unified_memory()` is false on a card, and main's MemAvailable block that this replaces never ran on a card, because dpct's integrated flag was always 0 before #3729.
- Other integrated GPUs (Lunar Lake, Arrow Lake, Meteor Lake), Windows.
- CUDA, HIP and Metal on their GPUs: no file of theirs changed; CI compiles them (above).

## Limits

- Only the B390 was measured, with one model.
- The headroom stays at 6 GiB. A desktop that holds more RAM gets a smaller cache, and the missing experts are read from the file.
- On this branch alone, prompts of about 6K tokens are around 15% slower than on main on this GPU; with #6455 they are about 48% faster.

Prepared with an AI coding assistant (Claude Code); every number above was measured on the machine named in Validation.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Mehr auf der Site

Links zu Install, Modellen, Releases.