#6654 [Engine] Periodically rebalance the resident expert cache as free VRAM changes
open · @leonardocapellaro · 0 comentarios · En GitHub
Descripción
## Problem After the picture reader returns its VRAM (#6653), the cache may still have fewer GPU slots than it started with. If the user then closes a browser or another GPU-heavy app, those slots stay empty until another resize request. A desktop application that increases its GPU use can also leave the server below its configured free-memory target. ## Proposal Offer an opt-in periodic VRAM check for the resident elastic expert cache. The server samples current GPU memory between requests, skips a check while the image encoder is active and calls the existing resize command when there is room for a segment or free memory has dropped materially below the target. It uses the startup reserve or the last manual reserve and keeps the startup cache capacity and reclaim boundary. A two-minute interval is configured for my local deployment. This prototype uses `STRATA_VRAM_REBALANCE_S` with zero as the default. The fork branch [resident-vram-periodic](https://github.com/leonardocapellaro/Strata/tree/resident-vram-periodic) builds on the prototype from #6653. The periodic change is in [commit 1c6d0d22b](https://github.com/leonardocapellaro/Strata/commit/1c6d0d22b). The 32 Vision and VRAM server tests passed, including a simulated drop in free memory, a skipped resize during Vision and the recovery of a partial segment. The timer can only reclaim backed segments and repopulate slots up to the capacity sized at model load. On the RTX 4090, the first scheduled check after loading the full model resized the cache from 8087 to 7898 slots in 24 ms and reported 1240 MiB free. Recovery after a simulated VRAM increase is covered by the server tests. Would the opt-in timer and scope be suitable as a follow-up to #6653?
En el sitio
Enlaces a install, modelos, releases.