Pull requests / #3030

#3030 fix(Metal): the prompt path no longer kills the engine on prompts > 6…

open · @DarkJoll · 0 comentarios · En GitHub

BenchmarksAMD / HIPNVIDIA / CUDAModels & quants

Descripción

fix(Metal): the prompt path no longer kills the engine on prompts > 64 tokens

Addresses #2671 - the Metal failure reported there (same machine family, M1 Max 64 GB; the
follow-up comment's diagnosis, `!m.borrowed` inside `Prefill::relayout`, matches what I measured).

## What this changes

On Metal, every prompt over 64 tokens kills the engine, in two ways:

1. a ~1.6k-token prompt: `lending the prompt path its slots failed: ... at most 0`, exit 1
   (the guard inside `Prefill::relayout`, as #2671 diagnosed);
2. a 10,797-token prompt: the read completes, then SIGABRT `a copy touches a device allocation
   of 21298.5 MB that was given back` inside `ExpertCache::fill_slot_queued`, exit -6. The Metal
   arena purges host allocations >= 32 MB (`metal/src/engine/window.cpp` `stream_arena` ->
   `purge_device_allocations(32<<20)`), and the post-read refill writes into that purged region
   through the slots the lend loop marked.

One root cause under both: on Metal the prompt's buffers live in the Metal arena (`window_ensure`),
not in the host region the host code believes it is borrowing, and the Metal window never reads the
host residency table. So the relayout has nothing to re-layout, and the lend has nothing to lend.

- `Prefill::relayout`: on Metal, a successful no-op that keeps the larger chunk count (the Metal
  buffers never shrink; a large `chunk()` is one `run()` per segment).
- the `lend` lambda: the slot-marking loop is guarded out, so `p.lent` stays empty and the refill
  after the read is a no-op instead of writing to purged host memory. `p.lent_chunk = want_p` stays.

## What this leaves alone

Every added line is inside a `STRATA_GPU_METAL` / `!STRATA_GPU_METAL` guard: the CUDA / HIP / SYCL
branch is the untouched pre-fix source, byte-identical by construction. The startup line "the prompt
path borrows N CUDA0 cache slots" still logs (cosmetic on Metal) and the sizing arithmetic is
untouched. No new flags, no behaviour change on any other backend.

## How I checked

Apple M1 Max 64 GB, macOS 27.0, engine 0.1.45-metal, test arena 8 GB (`STRATA_METAL_ARENA_GB=8`):

- pre-fix repro, both modes, deterministic: ~1.6k-token prompt -> "at most 0" exit 1;
  10,797-token prompt -> "given back" SIGABRT exit -6.
- post-fix: 4,643-token prompt -> prefill 85.3 tok/s, 512 tokens generated at 9.3 tok/s, engine
  alive after; 10,797-token prompt -> prefill 179.8 tok/s, 512 generated at 6.9 tok/s, engine alive
  after. Log over both: no "given back", no "at most 0", no lending error. `/health`: loaded.
- `ninja -C build-metal` full build 0 errors; `ctest` 1/1 (`runtime_hello`); `gr_parity` ALL PASS
  (prepared planes bitwise, rel-L2 <= 3.4e-08 on the rest); `prompt_parity` ALL PASS.

**Not checked:** the CUDA / HIP / SYCL builds cannot be compiled on a Mac (no toolchains). Per
AGENTS.md a build per backend is the check before merge; the guard makes the non-Metal branch the
untouched source, but I cannot claim the builds, only the construction.

## Numbers on the shipping configuration

Same machine, the default run (`run-iq3_s.sh`, IQ3_S, expert arena ~32.5 GB, 256K context):
prefill ~220 tok/s, decode ~20 tok/s, and long prompts complete with the engine staying up. These
are read off the server's own logs during normal use, not a controlled benchmark; the 8 GB-arena
numbers above are the controlled ones (the small arena is what makes its prefill slower).

En el sitio

Enlaces a install, modelos, releases.