Pull requests / #491

#491 docker: GGUF_DIR, RESIDENT_BUDGET_GIB and KV_STREAMING env vars; document the stop timeout

closed · @mad9home · 0 Kommentare · Auf GitHub

Setup & installNVIDIA / CUDAModels & quantsDocumentationLinux

Beschreibung

**What**

- `GGUF_DIR` passes setup's `--gguf-dir`, so a container can use model files the host already has. #252 and #444 answer this for the native setup; the entrypoint had no way to pass it, so `FAMILY=unsloth` downloaded 111 GB again.
- `RESIDENT_BUDGET_GIB` and `KV_STREAMING` pass `--resident-budget-gib` and `--kv-streaming`. Unset keeps setup's own defaults, so nothing changes for existing users.
- `docs/INSTALL.md`: the mounted GGUF folder has to be writable on the first start (the Unsloth SHA-256 result is kept as `<shard>.done` next to each shard; `:ro` works afterwards), and a note on the stop timeout.

**Why the stop timeout:** releasing ~56 GiB of page-locked RAM after SIGTERM takes ~20 s here, longer than Docker's 10 s default, so `docker stop` kills the server (exit 137). With `--stop-timeout 60` it exits cleanly (exit 0).

**Tested** on v0.1.35, RTX 3090, Linux, image built from this branch (`CUDA_ARCHITECTURES=86`, `BUILD_VISION=0`):

| Case | Result |
|---|---|
| First start, `GGUF_DIR=/gguf` (writable), `FAMILY=unsloth MODEL=UD-Q4_K_XL` | no download, config `--native /gguf/…`, healthy, answers |
| `RESIDENT_BUDGET_GIB=60 KV_STREAMING=on CONTEXT=65536` | `--resident-budget-gib 60`, `--kv-resident 32768` in the config |
| Second start, GGUF folder mounted `:ro` | setup skipped, healthy |
| `docker stop -t 10` / `--stop-timeout 60` | exit 137 after 21 s / exit 0 after 21 s |

No engine or setup.py change.

Mehr auf der Site

Links zu Install, Modellen, Releases.