Issues / #1275
#1275 Windows HIP: --prefill auto can pick a chunk whose VRAM borrow leaves too little for the verify captures — silent engine exit at serve init
open · @slogomansdad · 1 commentaires · Sur GitHub
Server & APIAMD / HIPModels & quantsWindows
Description
Engine 0.1.40 (also reproduced on 0.1.39), Windows 11, RX 9060 XT 16 GB (gfx1200), IQ2_XS, 128K context, `--kv-resident 32768`, expert cache auto (6,053 slots / 8.11 GiB), `--resident-budget-gib 12`. With plain `--prefill auto` the chunk selection picked **15,104 tokens**, borrowing 5,119 cache slots (6.85 GiB). After "313 MiB of VRAM free with everything loaded" the engine exited silently during the verify-window capture phase — no log line after the free-VRAM line; the server reports "the engine exited before it was ready". Reproduced on 4 consecutive starts. With `--prefill auto:8192` (borrow 2,925 slots / 3.90 GiB) the same machine starts and serves normally, so that's the workaround. Possibly related: #1250 (host OOM while allocating the page-locked IQ2_XS copy) — different allocation and different failure point, but both are Windows memory-pressure edge cases around the budget copy. Expected: the auto chunk rule could account for the post-borrow allocations (the verify captures are 57.2 MiB of device buffers, plus whatever else serve init needs), or clamp the borrow so post-borrow free VRAM stays above a floor.
Sur le site
Liens install, modèles, releases.