Pull requests / #3030
#3030 fix(Metal): the prompt path no longer kills the engine on prompts > 6…
open · @DarkJoll · 0 Kommentare · Auf GitHub
BenchmarksAMD / HIPNVIDIA / CUDAModels & quants
Beschreibung
fix(Metal): the prompt path no longer kills the engine on prompts > 64 tokens Addresses #2671 - the Metal failure reported there (same machine family, M1 Max 64 GB; the follow-up comment's diagnosis, `!m.borrowed` inside `Prefill::relayout`, matches what I measured). ## What this changes On Metal, every prompt over 64 tokens kills the engine, in two ways: 1. a ~1.6k-token prompt: `lending the prompt path its slots failed: ... at most 0`, exit 1 (the guard inside `Prefill::relayout`, as #2671 diagnosed); 2. a 10,797-token prompt: the read completes, then SIGABRT `a copy touches a device allocation of 21298.5 MB that was given back` inside `ExpertCache::fill_slot_queued`, exit -6. The Metal arena purges host allocations >= 32 MB (`metal/src/engine/window.cpp` `stream_arena` -> `purge_device_allocations(32<<20)`), and the post-read refill writes into that purged region through the slots the lend loop marked. One root cause under both: on Metal the prompt's buffers live in the Metal arena (`window_ensure`), not in the host region the host code believes it is borrowing, and the Metal window never reads the host residency table. So the relayout has nothing to re-layout, and the lend has nothing to lend. - `Prefill::relayout`: on Metal, a successful no-op that keeps the larger chunk count (the Metal buffers never shrink; a large `chunk()` is one `run()` per segment). - the `lend` lambda: the slot-marking loop is guarded out, so `p.lent` stays empty and the refill after the read is a no-op instead of writing to purged host memory. `p.lent_chunk = want_p` stays. ## What this leaves alone Every added line is inside a `STRATA_GPU_METAL` / `!STRATA_GPU_METAL` guard: the CUDA / HIP / SYCL branch is the untouched pre-fix source, byte-identical by construction. The startup line "the prompt path borrows N CUDA0 cache slots" still logs (cosmetic on Metal) and the sizing arithmetic is untouched. No new flags, no behaviour change on any other backend. ## How I checked Apple M1 Max 64 GB, macOS 27.0, engine 0.1.45-metal, test arena 8 GB (`STRATA_METAL_ARENA_GB=8`): - pre-fix repro, both modes, deterministic: ~1.6k-token prompt -> "at most 0" exit 1; 10,797-token prompt -> "given back" SIGABRT exit -6. - post-fix: 4,643-token prompt -> prefill 85.3 tok/s, 512 tokens generated at 9.3 tok/s, engine alive after; 10,797-token prompt -> prefill 179.8 tok/s, 512 generated at 6.9 tok/s, engine alive after. Log over both: no "given back", no "at most 0", no lending error. `/health`: loaded. - `ninja -C build-metal` full build 0 errors; `ctest` 1/1 (`runtime_hello`); `gr_parity` ALL PASS (prepared planes bitwise, rel-L2 <= 3.4e-08 on the rest); `prompt_parity` ALL PASS. **Not checked:** the CUDA / HIP / SYCL builds cannot be compiled on a Mac (no toolchains). Per AGENTS.md a build per backend is the check before merge; the guard makes the non-Metal branch the untouched source, but I cannot claim the builds, only the construction. ## Numbers on the shipping configuration Same machine, the default run (`run-iq3_s.sh`, IQ3_S, expert arena ~32.5 GB, 256K context): prefill ~220 tok/s, decode ~20 tok/s, and long prompts complete with the engine staying up. These are read off the server's own logs during normal use, not a controlled benchmark; the 8 GB-arena numbers above are the controlled ones (the small arena is what makes its prefill slower).
Mehr auf der Site
Links zu Install, Modellen, Releases.