Issues / #2477
#2477 8 GB card: setup's config leaves ~420 MiB of VRAM free, below the 512 MiB that `serve` warns about (#2065)
open · @Dongli99 · 0 commentaires · Sur GitHub
BenchmarksSetup & installServer & APINVIDIA / CUDAModels & quantsLinux
Description
RTX 3080 Laptop 8 GB, Linux, Strata 0.1.44.1 (`296b59d2`), Coder IQ1_M with setup's config (`--expert-cache auto --prefill auto --spec 4 --mtp ... --max-context 32768 --kv int8 --resident-experts`), nothing else on the GPU (15 MiB used before start). The start log says `expert cache auto: 1.99 GiB free, 700 MiB reserved (+218 MiB for the draft head) -> 441 slots`, then `expert cache 566 slots, 1.09 GiB of VRAM`. After loading, `serve` reports `423 MiB of VRAM free with everything loaded`. The first request prints `WARNING: only 423 MiB of VRAM free ... (below 512 MiB) ... Add --vram-reserve-mib 832`, and a later one warns at 291 MiB and suggests 960. Three starts gave 417-423 MiB free. IQ2_XS with its setup config (mapped): 481 MiB free after load, warnings at 481 and 335 MiB. We saw no paging stall: 0 failed requests in four bench runs, and no request decoded below 13 tok/s. On 0.1.43.1, `--vram-reserve-mib 832` gave 553 MiB free and 502 slots with no measurable decode change (not repeated on 0.1.44.1). Questions: should `auto` on a small card leave the room its own warning asks for, so setup's config starts without the warning? And is it expected that `auto` computes 441 slots but the cache ends with 566?
Sur le site
Liens install, modèles, releases.