Issues / #1275

#1275 Windows HIP: --prefill auto can pick a chunk whose VRAM borrow leaves too little for the verify captures — silent engine exit at serve init

open · @slogomansdad · 1 Kommentare · Auf GitHub

Server & APIAMD / HIPModels & quantsWindows

Beschreibung

Engine 0.1.40 (also reproduced on 0.1.39), Windows 11, RX 9060 XT 16 GB (gfx1200), IQ2_XS, 128K context,
`--kv-resident 32768`, expert cache auto (6,053 slots / 8.11 GiB), `--resident-budget-gib 12`.

With plain `--prefill auto` the chunk selection picked **15,104 tokens**, borrowing 5,119 cache slots
(6.85 GiB). After "313 MiB of VRAM free with everything loaded" the engine exited silently during the
verify-window capture phase — no log line after the free-VRAM line; the server reports "the engine exited
before it was ready". Reproduced on 4 consecutive starts.

With `--prefill auto:8192` (borrow 2,925 slots / 3.90 GiB) the same machine starts and serves normally,
so that's the workaround.

Possibly related: #1250 (host OOM while allocating the page-locked IQ2_XS copy) — different allocation
and different failure point, but both are Windows memory-pressure edge cases around the budget copy.

Expected: the auto chunk rule could account for the post-borrow allocations (the verify captures are
57.2 MiB of device buffers, plus whatever else serve init needs), or clamp the borrow so post-borrow free
VRAM stays above a floor.

Mehr auf der Site

Links zu Install, Modellen, Releases.