Pull requests / #1911

#1911 Expert cache: STRATA_VRAM_FREE=device reads the whole card's free VRAM through NVML (opt-in, Windows + NVIDIA)

open · @Xiaomi2008 · 0 commentaires · Sur GitHub

BenchmarksSetup & installServer & APIMulti-GPUAMD / HIPNVIDIA / CUDAModels & quantsDocumentationWindowsLinux

Description

## Title
Issue: none (searched the open issues and pull requests; nothing covers this)

## Summary
`STRATA_VRAM_FREE=device` (opt-in, Windows + NVIDIA): `--expert-cache auto` reads the free VRAM of the whole card through NVML, and takes the smaller of that and CUDA's own figure.

Under WDDM `cudaMemGetInfo` answers for the calling process only. Measured on an RTX 5090, driver 610.88: with 6 GiB written by another process, `cudaMemGetInfo` in a second process still read 30,994 MiB free where NVML read 24,348. So with the image encoder loaded (1,547 MiB here) the cache is sized from a figure that much too high, the check after the slots are written reads less than the reserve, and the cache is shrunk by guesses.

Six starts of one config on 0.1.40.3 and 0.1.41 (Swift 1.5 IQ3_XXS, 128K, the default 700 MiB reserve, image encoder on the GPU) ended with 9,002 to 14,454 slots and 93 MiB to 10 GiB of VRAM free.

The switch is used in three places: the sizing, the check after the slots are written, and the serve check's "VRAM free with everything loaded" figure. `nvml.dll` is loaded from the system directory at first use and the device is found by its PCI bus id. One line at first use prints both figures.

## What changed
- src/core/expert_cache.cpp: `device_free_bytes()` takes the smaller of CUDA's figure and NVML's when the variable is set (a new `nvml_device_free`, Windows CUDA builds only).
- include/strata/core/expert_cache.hpp: the comment on `device_free_bytes()`.
- src/program/generate.cpp: the serve check's free figure goes through the same function (one line).
- docs/DETAILS.md: one entry with the measurements.

## Extra Notes
### Measured

This branch (0.1.42 + this change), RTX 5090, Core i7-13700K, 64 GB, Windows 11, Swift 1.5 IQ3_XXS, 128K, image encoder on the GPU, three interleaved server starts each:

| | slots | shrink steps | VRAM free with everything loaded |
|---|---|---|---|
| default (700 MiB reserve) | 13,543 | 2 | 336 / 602 / 602 MiB |
| `STRATA_VRAM_FREE=device`, `--vram-reserve-mib 1500` | 13,102 - 13,130 | 1 | 1,107 / 1,225 / 1,288 MiB |

At the sizing NVML read 23,695 MiB free where CUDA read 25,458. On 0.1.41 with the switch: one pass, 13,007 to 13,188 slots and 973 to 1,392 MiB free over seven starts.

Why the headroom matters here: a run that filled the card to 0 MiB free decoded 80-115 tok/s where the same forced text ran at 205-230 tok/s with 900 MiB or more free.

### What it leaves alone

- Unset, `device_free_bytes()` returns what it did, and the serve check's figure is the same value (`min` of the same number). I did not run the byte-identity tests; this is from the code path: the new branch is behind the environment variable.
- The cache size changes no output bit.
- Linux, HIP and SYCL: the new code is compiled for Windows CUDA builds only (`STRATA_EC_NVML_FREE`).
- The shrink loop itself is unchanged.

### Not tested

- Built for CUDA only: sm_120, CUDA 13.0.88, MSVC 14.50 with nvcc's `--allow-unsupported-compiler` (this PC has no Visual Studio 2022). **HIP and SYCL were not compiled.** `expert_cache.cpp` and `generate.cpp` are built by both; the HIP build compiles none of the new branch (it sits under `!STRATA_EC_NO_VMM`), SYCL I did not look at.
- One machine, one card, one driver. No multi-GPU run (the device is matched by PCI bus id, which I could only exercise with one card).
- Cards where another process holds no VRAM show no difference, by construction.

Docs: one entry in `docs/DETAILS.md`.

### In use, with the rest of a tuned config (context)

0.1.42 built from source, through the server, two interleaved rounds of the same requests on this PC: setup's arguments ended with 458 / 561 MiB of VRAM free with everything loaded, a config with `STRATA_VRAM_FREE=device` and `--vram-reserve-mib 1500` with 1,039 / 1,083 MiB. That config also changes the CPU pool layout and other switches, and its speed table is in #1912; none of the speed difference there is this change, which only decides how much VRAM the cache takes.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Sur le site

Liens install, modèles, releases.