#3729 [Engine] SYCL: flag Intel iGPUs as integrated so the expert cache is sized from MemAvailable

open · draft · @sashebalchev · 0 comentarios · En GitHub

BenchmarksServer & APIAMD / HIPNVIDIA / CUDAModels & quantsLinux

Descripción

## Problem and result
`get_device_info()` in `sycl/include/dpct/device.hpp` never calls `set_integrated()`, so `get_integrated()` is 0 on every SYCL device. On a shared-memory iGPU `device_free_bytes()` (`sycl/src/core/expert_cache.cpp`) therefore never takes its `MemAvailable` branch and sizes the expert cache from the tiny free "card" memory.

On an Arc B390 (Panther Lake, xe) that gave **82 slots (0.16 GiB) even with an explicit `--expert-cache 12288`**, all 12,206 other experts mirrored in pinned host memory and read over PCIe. With the one-line fix the same command gets **12,288 slots (23.42 GiB)**. Same bug as reported for Meteor Lake in #2282 (bug 1, same fix); it was closed in the tracker reset and `set_integrated()` still has no caller on `main`.

## Related discussion
#2282 (bug 1; this PR is only that part). Also touches the `MemAvailable` dependency described in #2412. No default or scope change: it only makes the existing integrated-GPU path reachable on SYCL.

## Approach
Set the flag from `sycl::aspect::ext_oneapi_is_integrated_gpu`, next to the other `prop.set_*` calls in `get_device_info()`. Because `sycl/include/dpct/` is dpct output that `migrate.sh` / `normalize.sh` regenerate, the same line is also an idempotent entry in `sycl/tools/fixups.py` (like the `util.hpp` fix), so it survives the next re-migration. Checked: running `fixups.py` on the unpatched header gives exactly the committed header, and a second run changes nothing.

Side effect: `ggml_cuda_host.dp.cpp:58` copies the flag into `d.integrated`, which has no reader in the SYCL build (`grep -rn integrated sycl/src sycl/include` finds only the definition, that writer and `expert_cache.cpp:112`). The behaviour change is confined to `device_free_bytes()`, and per device and on Linux only. Older runtimes without the aspect report 0, i.e. the old behaviour.

## Validation
Hardware: Framework 13 Pro, Core Ultra X7 358H, Arc B390 (xe), 64 GB shared, Linux 7.3; Coder IQ1_M, engine 0.1.46.3, `--serve --expert-cache 12288 --stream-experts --mtp --max-context 32768 --kv int8`, started through `sycl/serve/strata-sycl.sh`. Only the rebuilt binary differs between the two runs (log: `strata-coder-iq1_m.log`, 2026-10-21 10:27 vs 10:33).

**Passed:**
- Build in the `strata-sycl-dev` container: `bash sycl/tools/build.sh strata`, 75/75 targets, 0 errors (revision with the header edit). After adding the `fixups.py` entry and dropping a redundant `? 1 : 0`: `icpx -fsycl -std=c++20 -fsyntax-only` on a TU including `dpct/dpct.hpp` and calling `get_device_info()`, RC=0; the full engine was not rebuilt at this revision.
- Same start command and flags before and after; only the rebuilt binary differs. Engine log of both starts (same day, 10:27 and 10:33):
  - Before: `expert cache 82 slots, 0.16 GiB of VRAM`, `12206 of 12206 experts missing from VRAM ... mirrored in pinned host memory`.
  - After: `expert cache 12288 slots, 23.42 GiB of VRAM`, profile fill 6.0 s, `token graph hit path: 12288 resident experts`. Min `MemAvailable` during start 24.2 GiB, no xe errors.
- Every request I made, individually (chat completion "Write a Python function is_prime(n). Only the code.", thinking off, max 200 tokens, fresh server, no warm-up):

  | build | request | prompt | decode | drafts accepted |
  |---|---|---|---|---|
  | before | 1 (cold) | 24 tok in 26.8 s | 6.8 tok/s (91 tok) | 60 of 66 |
  | after | 1 (cold) | 7.2 s | 14.8 tok/s | 75 of 86 |
  | after | 2 | 0.1 s (prefix reused) | 21.2 tok/s | 58 of 70 |

  Compare cold with cold (rows 1 and 2). I have no second "before" request.
- A coding agent session of about 8,000 prompt tokens then ran normally against the patched build (17.5 and 22.4 tok/s on its two requests).

**Failed:** none observed.

**Not run:**
- CUDA, HIP and Metal builds: not touched (change is under `sycl/` only).
- Any other Intel iGPU. Meteor Lake evidence is from #2282, not from me.
- Repetitions: the guide asks for at least three runs per configuration and `tools/ab_engine.py` / `strata_bench.py --compare` for before/after. I have one cold request on the unpatched build and two on the patched one, so the tok/s figures are indicative and no speedup is claimed for other machines. The slot count is the actual finding. I can add an `ab_engine.py` run if a maintainer wants it.
- No regression test: the check needs an iGPU.

## Limits
- Makes the existing integrated-GPU path reachable on SYCL, which also makes bug 2 of #2282 reachable on other iGPUs: the cache then becomes one ~23 GiB allocation, and `expert_cache.cpp:666` already notes that an Arc A750 caps one allocation at 4 GiB unless `UR_L0_ENABLE_RELAXED_ALLOCATION_LIMITS=1`. My B390 did not hit it. Bug 3 (`STRATA_WINDOW_SYNC` on xe) is also not included.
- The sizing relies on `MemAvailable` less `STRATA_UMA_HEADROOM_GIB` (default 6). Here `--expert-cache 12288` took 23.42 GiB against a minimum `MemAvailable` of 24.2 GiB during start, so the rest of the run used about 0.8 GiB of that headroom. The TTM page pool described in #2412 can lower `MemAvailable` before a start.
- Not addressed: the TTM page-pool effect.

<details><summary>Engine log, both starts (paths shortened)</summary>

```
strata generate: profile /work/strata/data/expert-profile-coder.bin: 12288 ranked pairs, built for 12288 slots
strata generate: PLE on, table 320001536 rows of /work/Strata-data/models/coder-IQ1_M/Qwen3.8-Flash-Next-GSQ-RCO-IQ1_M-00002-of-00002.gguf
strata mtp: draft layer loaded, 839 MiB of VRAM (experts 675, dense 111), files read in 0.30 s (2578 MiB/s)
strata: STRATA_Q6_PLANAR: the output head (248320 x 2560) is stored as 16-byte-aligned planes (same bytes)
strata generate: experimental native Q5_K head, 521472000 bytes, in 0.3 s
strata generate: GPU 0: Intel(R) Arc(TM) B390 GPU, compute capability 30.0
strata generate: experts streamed from the GGUF on demand (--stream-experts): no host arena; every expert must fit the VRAM cache
strata generate: 9 pool workers on logical processors 0,1,3,4,5,6,7,8,9, the host thread on 2 (draining too) (--host-core first)
strata generate: expert cache 82 slots, 0.16 GiB of VRAM; policy is
strata generate: the GPU computes the experts in the cache; it rounds differently from the CPU,
                 so a reply can differ slightly from a run without the cache (same quality:
                 bench/results/2026-09-27-cache-parity).
                 PROFILE, ranked by routing frequency, no eviction.
strata generate: profile fill: 0.16 GiB in 0.1 s (2.31 GB/s, pipelined)
strata generate: pre-filled 82 of 82 slots from the profile in 0.1 s (2305 MB/s); slot 0 verified
strata generate: 12206 of 12206 experts missing from VRAM (+ 0 in the prompt loan's slots) mirrored in pinned host memory (23.26 GiB, 14.8 s); the GPU reads them over PCIe
strata generate: adaptive swaps off: with STRATA_VERIFY_NO_HOST an expert they move out of VRAM would be neither there nor mirrored
strata generate: R4 hit path ON - resident experts are computed on the GPU
strata generate: 9 expert-pool workers + the host thread
strata generate: session is up (engine 0.1.46.3)
strata generate: token graph hit path: 82 resident experts, decided on the device
strata serve: the prompt path allocates its own buffers (too few cache slots to borrow)
...
strata generate: profile /work/strata/data/expert-profile-coder.bin: 12288 ranked pairs, built for 12288 slots
strata generate: PLE on, table 320001536 rows of /work/Strata-data/models/coder-IQ1_M/Qwen3.8-Flash-Next-GSQ-RCO-IQ1_M-00002-of-00002.gguf
strata mtp: draft layer loaded, 839 MiB of VRAM (experts 675, dense 111), files read in 0.57 s (1385 MiB/s)
strata: STRATA_Q6_PLANAR: the output head (248320 x 2560) is stored as 16-byte-aligned planes (same bytes)
strata generate: experimental native Q5_K head, 521472000 bytes, in 0.4 s
strata generate: GPU 0: Intel(R) Arc(TM) B390 GPU, compute capability 30.0
strata generate: experts streamed from the GGUF on demand (--stream-experts): no host arena; every expert must fit the VRAM cache
strata generate: 9 pool workers on logical processors 0,1,3,4,5,6,7,8,9, the host thread on 2 (draining too) (--host-core first)
strata generate: expert cache 12288 slots, 23.42 GiB of VRAM; policy is
strata generate: the GPU computes the experts in the cache; it rounds differently from the CPU,
                 so a reply can differ slightly from a run without the cache (same quality:
                 bench/results/2026-09-27-cache-parity).
                 PROFILE, ranked by routing frequency, no eviction.
strata generate: profile fill: 23.42 GiB in 6.0 s (4.18 GB/s, pipelined)
strata generate: pre-filled 12288 of 12288 slots from the profile in 6.0 s (4177 MB/s); slot 0 verified
strata generate: R4 hit path ON - resident experts are computed on the GPU
strata generate: 9 expert-pool workers + the host thread
strata generate: session is up (engine 0.1.46.3)
strata generate: token graph hit path: 12288 resident experts, decided on the device
strata serve: the prompt path allocates its own buffers (too few cache slots to borrow)
strata serve: a 2048-token chunk's prompt buffers need 1130 MiB on CUDA0, 981 MiB free: 512-token chunks
```
</details>

En el sitio

Enlaces a install, modelos, releases.