Pull requests / #2345
#2345 fix(metal): prompt buffer fallback when arena chunks are too small (#2323)
open · @kingdonb · 0 Kommentare · Auf GitHub
BenchmarksModels & quantsDocumentation
Beschreibung
Completes #2323 for small-memory Macs. #2323's reporter ran a 96 GB M2 Max, where the arena's chunks are several GB and the prompt's 1.60 GB of buffers always fit. On a 24 GB M2 the arena shrinks at start (here: chunks of 0.8–1.0 GB), every chunk is skipped by the bytes + (64 MB) check in ExpertStream::region, and every prompt stops with metal prompt: no arena region for the prompt buffers.
The standalone-mmap fallback ("the arena is small: the prompt's … buffers are their own allocation", from the small-arena work) sat inside the chunk loop behind the slot-class check, so it was only reached when a chunk was big enough. This moves it into one function both call sites use: a fitting chunk whose slots a class cannot spare (behavior unchanged), and no fitting chunk at all (was a hard stop, now the same standalone allocation).
Changes: metal/src/engine/stream.cpp (+ one private helper, declared in stream.h), one sentence in docs/MAC.md. Leaves alone: the fitting-chunk carve, the slot-class rule, arena sizing, decode paths; only metal/ is touched (Metal-only files; built on the M2 with Apple clang 21 / metal-cpp_26).
No arena that could host the prompt region before behaves differently — the new path only runs where region() returned nullptr and the first prompt failed. Verified on a 24 GB M2 (macOS 26.6.2, Qwen3.8-Flash-Next GSQ-RCO-Q2_0, 31.64 GiB .experts): the fallback line replaces the error, prompts run end-to-end (3.7–4.1 tok/s steady in web chat, multi-turn, no crashes).
Not tested here: the M4/M5 (not on hand — the fitting-chunk and tight-slot paths are statement-identical to 0.1.44.1, so their numbers should stand); prompt tok/s before/after (the fallback replaces a hard stop, so there is no "before" to compare); and the memory-pressure story of the extra ~1.6 GB beside a shrunken arena (nothing swapped in our runs; a proper measurement is wanted).Mehr auf der Site
Links zu Install, Modellen, Releases.