Issues / #1146

#1146 LLM Server feature request: "idle_load: true" parameter

open · @KoenKeunen · 2 Kommentare · Auf GitHub

Beschreibung

Strata works great as a network LLM server but lacks an important startup parameter: **idle_load: true**. In other words: to be able to start the server without loading the LLM into VRAM/DDR, and only start loading into vram the user requests for it. This makes it possible to start the server in the background and it does not utilize system resources until the moment the LLM is really necessary.

It's great "--idle-unload" "600" is possible already to offload the model after 10 minutes of inactivity, but it would be great to also start Strata with offloaded model by default.

Mehr auf der Site

Links zu Install, Modellen, Releases.