Pull requests / #2190

#2190 sycl: size the expert cache from the DRM fdinfo when xe reports 0 free card memory (Arc A770, #1545)

open · @aslater3 · 0 comments · View on GitHub

BenchmarksAMD / HIPNVIDIA / CUDAModels & quantsWindowsLinux

Description

## Title
Issue: #1545 (Arc A770 on `xe`); related to #1606

## Summary
On an Arc A770 16 GB on the `xe` driver, Level Zero has no `ext_intel_free_memory`, so `get_memory_info` returns 0 free bytes (with the right total). `device_free_bytes()` kept the 0 — the #1440 fdinfo path only lowers a figure that is too high — so `--expert-cache auto` sized the cache from 0 bytes and a config with `--spec` could not start. This reads "0 free with a known total" as unknown and, on Linux, uses the card's total less this process's DRM fdinfo figure (the same source the A750/i915 path already uses). Any driver that reports a non-zero free figure is unaffected; off Linux the 0 stands.

## What changed
- `sycl/src/core/expert_cache.cpp` (`device_free_bytes`): when the driver reports 0 free and a non-zero total, on Linux, `free = total - own_drm_local_bytes()`, with a one-time message. Nothing else.

## How it was checked (Arc A770 16 GB, `xe`, Linux 7.0, oneAPI 2026.1, Ryzen 7 3800X, 30 GB RAM; JIT build of `a0116b27` with and without this commit; same day)
Coder IQ1_M, `--expert-cache auto --resident-experts --max-context 131072 --kv int8 --kv-resident 20480 --vram-reserve-mib 1200 --prefill-borrow --prefill auto --spec 4 --spec-min-p 0.5 --mtp ...`, an 8K-token prompt:

- `main` (`a0116b27`):
  ```
  get_memory_info: ext_intel_free_memory is not supported.
  strata generate: expert cache auto: 0.00 GiB free, 1200 MiB reserved (+5 MiB for the draft head) -> 0 slots
  strata generate: no VRAM is left for the expert cache: it needs at least 286 slots (726 MiB), about 1932 MiB more than this card has free. ...
  strata generate: --spec needs the device residency table (--expert-profile, --expert-cache and the token graph)
  ```
  exit code 2.
- With this commit:
  ```
  strata: the driver reports no free card memory (xe without ext_intel_free_memory); sizing from the DRM fdinfo (this process holds 5.29 GiB)
  strata generate: expert cache auto: 9.82 GiB free, 1200 MiB reserved (+5 MiB for the draft head) -> 3486 slots
  ```
  the 8K prompt is read (542-555 tok/s) and decoding runs.

Not built for CUDA/HIP (the file is SYCL-only). Not tested on Windows (unchanged there by construction) or with two cards on `xe`.

## Extra Notes
- @flare04 reported carrying a "device_free_bytes() host-mirror sizing fix" for the same A770/xe case in #1545; this may overlap with it.
- Upstream's own `--stream-experts` config (setup_intel) does not fit this 30 GB machine for Coder IQ1_M at 128K (the mirror needs ~23.5 GB pinned; 15.7 GB fit, then `REFUSED: 2137 experts are neither in VRAM nor mirrored`), which is why the A770 here runs `--resident-experts`. Separate from this PR; it will be in the A770 benchmark report.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Related on strata.com

Editorial links to help you install, pick models, or read release notes — not part of the upstream thread.