Pull requests / #2836

#2836 Windows: make device_free_bytes() unified-memory aware on APUs (Strix Halo gfx1151)

open · @moweishun · 0 Kommentare · Auf GitHub

BenchmarksSetup & installAMD / HIPNVIDIA / CUDAModels & quantsWindowsLinux

Beschreibung

# Windows: `device_free_bytes()` is not unified-memory aware, so the auto expert cache cannot size on Strix Halo (gfx1151)

## Summary

On Windows, an APU with unified memory (Ryzen AI Max+ 395 / Radeon 8060S, `gfx1151`) reports a tiny
"free VRAM" figure through `cudaMemGetInfo` — essentially the free space of the BIOS's dedicated
carve-out, often well under 2 GiB. `device_free_bytes()` (`src/core/expert_cache.cpp`) only applies
its unified-memory correction (`MemAvailable` minus headroom) under `#if defined(__linux__)`, so on
Windows the auto expert cache sizes from the small figure and refuses to start
("no VRAM is left for the expert cache"), and an explicit `--expert-cache N` is silently shrunk to
0 slots by the sizing pass ("pre-filled 0 of 0 slots from the profile", no "expert cache N slots"
line).

This PR extends the same unified-memory correction to Windows: when the current device reports
integrated graphics, the free figure is raised to `GlobalMemoryStatusEx().ullAvailPhys` minus the
same `STRATA_UMA_HEADROOM_GIB` headroom (default 6 GiB). The Linux path is unchanged, and discrete
GPUs are unchanged (the integrated check is per device, as before).

## What it changes

- `src/core/expert_cache.cpp`, `device_free_bytes()`: the integrated-GPU detection moves out of the
  `__linux__`-only block (HIP exposes `cudaDevAttrIntegrated`), and a `_WIN32` branch adds the
  `GlobalMemoryStatusEx` path beside the Linux `/proc/meminfo` path. The
  `STRATA_UMA_HEADROOM_GIB` knob, its default of 6, and its semantics are shared.
- `#include <windows.h>` under `_WIN32` for `GlobalMemoryStatusEx`.

## What it leaves alone

- The Linux `/proc/meminfo` path, the gfx906 compat exclusion, the headroom default and env knob.
- Discrete GPUs on every platform (`unified_memory` false → plain `cudaMemGetInfo`).
- The SYCL port's own copy (`sycl/src/core/expert_cache.cpp`) — not touched.

## Measured on

Ryzen AI Max+ 395 (Radeon 8060S, `gfx1151`, wave32), Windows 11 (build 26100), Adrenalin driver
32.0.31041.1004, self-built HIP engine from this tree (`tools/hip/build_windows.bat`,
`STRATA_HIP_ARCHS=gfx1151`, ROCm 10.2.0a20260930 wheels), IQ3_S, 200K context, MTP on.

Free figure seen by the engine (after the session's allocations), by BIOS carve-out, before the fix:

| carve-out | `device_free_bytes()` | result |
|---|---|---|
| 0.5 GB | ~0 | auto: refuse; forced: shrunk to 0 slots |
| 16 GB | 0.37 GiB | auto: refuse (needs 241 slots / 611 MiB at 32K) |
| 32 GB | ~1.1 GiB | only 32K context fits (438 slots); 200K refuses |
| 64 GB | 6.69 GiB | 200K sizes 19339 slots — but see the caveats below |

After the fix (16 GB carve-out, 200K): auto sizes 19339 slots (36.84 GiB), pre-fills all of them
from the profile (6.5 s), serves at 40-45 tok/s decode (96.9% cache hit rate), where every
configuration refused to start or shrank to 0 slots before.

## Caveats, stated plainly

- On this chip the dedicated carve-out is smaller than the cache either way, so the cache lands in
  the driver's shared pool (WDDM GTT backed by the same LPDDR5X). The server's
  "0 MiB of VRAM free with everything loaded - LOW" warning still prints (the reserve check reads
  the same function, which now reports the large figure and never triggers the shrink loop).
  Measured no stalls across benchmark runs; a follow-up that also accounts for the carve-out in the
  shrink loop may be worth it, but it is deliberately not in this PR.
- The cache allocation is charged to the system commit on Windows (issue #60's note), so a big
  cache wants a page file with room. `tools/hip/build_windows.bat`'s engine printed that note
  during a failed open before the fix; with the fix the auto path converged on the first open.

## How it was checked

- Built the HIP engine for gfx1151 with the patch and ran setup + serving end to end (the numbers
  above). Without the patch, the same tree reproduces the refusals in the table.
- `src/core/expert_cache.cpp` is also compiled by the CUDA build; on Windows CUDA there is no
  integrated AMD GPU, and on Linux the CUDA build already took the `__linux__` path (unchanged).
  Not built here (no NVIDIA GPU on this PC) — the CUDA path's behaviour for a discrete GPU is
  unchanged by construction (the new code only runs when `unified_memory` is true).

Mehr auf der Site

Links zu Install, Modellen, Releases.