Pull requests / #491
#491 docker: GGUF_DIR, RESIDENT_BUDGET_GIB and KV_STREAMING env vars; document the stop timeout
closed · @mad9home · 0 comentários · No GitHub
Setup & installNVIDIA / CUDAModels & quantsDocumentationLinux
Descrição
**What** - `GGUF_DIR` passes setup's `--gguf-dir`, so a container can use model files the host already has. #252 and #444 answer this for the native setup; the entrypoint had no way to pass it, so `FAMILY=unsloth` downloaded 111 GB again. - `RESIDENT_BUDGET_GIB` and `KV_STREAMING` pass `--resident-budget-gib` and `--kv-streaming`. Unset keeps setup's own defaults, so nothing changes for existing users. - `docs/INSTALL.md`: the mounted GGUF folder has to be writable on the first start (the Unsloth SHA-256 result is kept as `<shard>.done` next to each shard; `:ro` works afterwards), and a note on the stop timeout. **Why the stop timeout:** releasing ~56 GiB of page-locked RAM after SIGTERM takes ~20 s here, longer than Docker's 10 s default, so `docker stop` kills the server (exit 137). With `--stop-timeout 60` it exits cleanly (exit 0). **Tested** on v0.1.35, RTX 3090, Linux, image built from this branch (`CUDA_ARCHITECTURES=86`, `BUILD_VISION=0`): | Case | Result | |---|---| | First start, `GGUF_DIR=/gguf` (writable), `FAMILY=unsloth MODEL=UD-Q4_K_XL` | no download, config `--native /gguf/…`, healthy, answers | | `RESIDENT_BUDGET_GIB=60 KV_STREAMING=on CONTEXT=65536` | `--resident-budget-gib 60`, `--kv-resident 32768` in the config | | Second start, GGUF folder mounted `:ro` | setup skipped, healthy | | `docker stop -t 10` / `--stop-timeout 60` | exit 137 after 21 s / exit 0 after 21 s | No engine or setup.py change.
No site
Links install, modelos, releases.