#6653 [Engine] Allow an opt-in resident expert cache to lend VRAM to vision
open · @leonardocapellaro · 0 comments · View on GitHub
Multi-GPUAMD / HIPNVIDIA / CUDAModels & quants
Description
## Problem With `--resident-experts` and on-demand GPU vision, the engine refuses the VRAM resize command because some experts exist only in the GPU cache. It keeps 2048 MiB reserved for the picture reader at startup. Users with tight VRAM can benefit from using that room for experts until a picture arrives. ## Proposed behavior Offer an opt-in bounded resident cache tail backed by pinned host RAM. On an image request the existing VRAM command may release backed GPU segments. Restore the slots after the reader stops. Keep a configurable desktop reserve and refuse a shrink or prompt loan when an affected expert has no host copy. The tradeoff is additional pinned RAM from startup, including room for prompt borrowing after shrinking. ## Evidence On Fedora 45 with an RTX 4090, Strata 0.2.19 / engine 0.1.65, Qwen3.8 Flash Next UD-IQ3_XXS v3.1 and BF16 vision, an experimental CUDA build started with 8044 GPU slots. An image request reduced the cache to 7107 slots, loaded the reader on the GPU and correctly identified a red image. GPU segment and host-backing tests passed. After the reader stopped at 120 seconds, the cache grew back to 7898 slots with 1118 MiB reported free. The resident log reported zero file-backed blobs, while a separate prefill counter reported 16401.6 MB of GGUF expert reads. That discrepancy remains under investigation. HIP, SYCL and Metal have not been tested. ## Prototype code The tested prototype is available in [the fork branch](https://github.com/leonardocapellaro/Strata/tree/resident-vram-elastic), with [the changes against the upstream base](https://github.com/Niko1221/Strata/compare/c6f6427d9de9016fd92d13e29740f65625261df7...leonardocapellaro:Strata:resident-vram-elastic). Would this opt-in approach and scope be suitable for a contribution? I can submit a pull request after scope agreement, with its limits and validation results.
Related on this site
Quick links to related pages.