Issues / #1146
#1146 LLM Server feature request: "idle_load: true" parameter
open · @KoenKeunen · 2 评论 · 在 GitHub 查看
描述
Strata works great as a network LLM server but lacks an important startup parameter: **idle_load: true**. In other words: to be able to start the server without loading the LLM into VRAM/DDR, and only start loading into vram the user requests for it. This makes it possible to start the server in the background and it does not utilize system resources until the moment the LLM is really necessary. It's great "--idle-unload" "600" is possible already to offload the model after 10 minutes of inactivity, but it would be great to also start Strata with offloaded model by default.
站内延伸阅读
链到安装、模型与版本说明,便于 SEO/GEO,非官方 issue 正文。