#6182 [Engine] Metal: --prefill auto sends 1024-token calls, so the hostpipe never overlaps the GPU (~10-12% slower prefill)
open · @ain-soph · 2 comments · View on GitHub
BenchmarksAMD / HIPNVIDIA / CUDAModels & quants
Description
Problem: On Metal, `--prefill auto` sends the engine 1024-token calls. Each call is then a single Metal prompt chunk, so the hostpipe never runs: the next chunk's embedding and PLE rows are computed on the CPU while the GPU waits. Prefill with `auto` is about 10–12% slower than a manual `--prefill 4096` with `STRATA_METAL_PROMPT_CHUNK=1024`. Cause: in `src/program/generate.cpp`, `auto` takes the CUDA branch "nothing lendable: small buffers of its own" (`o.prefill_chunk = 1024`). The Metal prompt path never borrows expert-cache slots, so this branch is always taken there. Environment: `main` c77572806, Metal engine build (`metal/`, `STRATA_GEOMETRY=flash_next`), macOS on an M5 Ultra with 96 GB, Qwen3.8-Flash-Next AD-4.27 resident pack, `--expert-cache 64 --spec 3 --mtp … --max-context 262144 --conversation-cache-mib 1024`, server `parallel: 2`. Reproduce: start the server with `--prefill auto`, send a ~26K and a ~116K token prompt after a short warm-up request, and read the prefill tok/s. Repeat with `--prefill 4096` and `STRATA_METAL_PROMPT_CHUNK=1024`. Actual (`main`, warm, two rounds): 26K 2,086 / 2,083 tok/s; 116K 1,997 / 1,997 tok/s. Manual `--prefill 4096` + `STRATA_METAL_PROMPT_CHUNK=1024` on the same build: 116K 2,237 tok/s, with the same temp-0 answer to a 38K-token prompt. Proposed: on Metal, `auto` keeps the 1024-row prompt chunks and hands each call 4 of them (`prompt_chunk_for_auto()` in `metal/src/engine/window.cpp`), so the hostpipe overlaps the GPU. CUDA, HIP and SYCL keep the current behaviour. A draft PR with this change and its before/after numbers follows: 26K 2,085 → 2,284 tok/s and 116K 1,997 → 2,240 tok/s, identical temp-0 output. This changes what the `auto` default does on Metal, so I'd like to agree the scope here first. Reproduction status: reproduced locally (logs available). This report was prepared with an AI coding agent (Claude Code); I checked the numbers against the run logs.
Related on this site
Quick links to related pages.