Issues / #1146
#1146 LLM Server feature request: "idle_load: true" parameter
open · @KoenKeunen · 2 Kommentare · Auf GitHub
Beschreibung
Strata works great as a network LLM server but lacks an important startup parameter: **idle_load: true**. In other words: to be able to start the server without loading the LLM into VRAM/DDR, and only start loading into vram the user requests for it. This makes it possible to start the server in the background and it does not utilize system resources until the moment the LLM is really necessary. It's great "--idle-unload" "600" is possible already to offload the model after 10 minutes of inactivity, but it would be great to also start Strata with offloaded model by default.
Mehr auf der Site
Links zu Install, Modellen, Releases.