Pull requests / #2190
#2190 sycl: size the expert cache from the DRM fdinfo when xe reports 0 free card memory (Arc A770, #1545)
open · @aslater3 · 0 comments · View on GitHub
BenchmarksAMD / HIPNVIDIA / CUDAModels & quantsWindowsLinux
Description
## Title Issue: #1545 (Arc A770 on `xe`); related to #1606 ## Summary On an Arc A770 16 GB on the `xe` driver, Level Zero has no `ext_intel_free_memory`, so `get_memory_info` returns 0 free bytes (with the right total). `device_free_bytes()` kept the 0 — the #1440 fdinfo path only lowers a figure that is too high — so `--expert-cache auto` sized the cache from 0 bytes and a config with `--spec` could not start. This reads "0 free with a known total" as unknown and, on Linux, uses the card's total less this process's DRM fdinfo figure (the same source the A750/i915 path already uses). Any driver that reports a non-zero free figure is unaffected; off Linux the 0 stands. ## What changed - `sycl/src/core/expert_cache.cpp` (`device_free_bytes`): when the driver reports 0 free and a non-zero total, on Linux, `free = total - own_drm_local_bytes()`, with a one-time message. Nothing else. ## How it was checked (Arc A770 16 GB, `xe`, Linux 7.0, oneAPI 2026.1, Ryzen 7 3800X, 30 GB RAM; JIT build of `a0116b27` with and without this commit; same day) Coder IQ1_M, `--expert-cache auto --resident-experts --max-context 131072 --kv int8 --kv-resident 20480 --vram-reserve-mib 1200 --prefill-borrow --prefill auto --spec 4 --spec-min-p 0.5 --mtp ...`, an 8K-token prompt: - `main` (`a0116b27`): ``` get_memory_info: ext_intel_free_memory is not supported. strata generate: expert cache auto: 0.00 GiB free, 1200 MiB reserved (+5 MiB for the draft head) -> 0 slots strata generate: no VRAM is left for the expert cache: it needs at least 286 slots (726 MiB), about 1932 MiB more than this card has free. ... strata generate: --spec needs the device residency table (--expert-profile, --expert-cache and the token graph) ``` exit code 2. - With this commit: ``` strata: the driver reports no free card memory (xe without ext_intel_free_memory); sizing from the DRM fdinfo (this process holds 5.29 GiB) strata generate: expert cache auto: 9.82 GiB free, 1200 MiB reserved (+5 MiB for the draft head) -> 3486 slots ``` the 8K prompt is read (542-555 tok/s) and decoding runs. Not built for CUDA/HIP (the file is SYCL-only). Not tested on Windows (unchanged there by construction) or with two cards on `xe`. ## Extra Notes - @flare04 reported carrying a "device_free_bytes() host-mirror sizing fix" for the same A770/xe case in #1545; this may overlap with it. - Upstream's own `--stream-experts` config (setup_intel) does not fit this 30 GB machine for Coder IQ1_M at 128K (the mirror needs ~23.5 GB pinned; 15.7 GB fit, then `REFUSED: 2137 experts are neither in VRAM nor mirrored`), which is why the A770 here runs `--resident-experts`. Separate from this PR; it will be in the A770 benchmark report. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
Related on strata.com
Editorial links to help you install, pick models, or read release notes — not part of the upstream thread.