#6183 [Engine] Metal: --prefill auto hands each call 4 prompt chunks so the hostpipe overlaps the GPU
open · draft · @ain-soph · 0 commentaires · Sur GitHub
BenchmarksAMD / HIPNVIDIA / CUDAModels & quants
Description
## Problem and result On the resident Metal server, `--prefill auto` falls back to 1,024-token calls, leaving the host pipeline one chunk at a time. This draft gives the resident host pipeline up to four 1,024-row chunks per call. It reaches the existing manual setting on the M5 Ultra. The default change is still for discussion in #6182. ## Approach After the Metal window is ready and its resident/streaming mode is known, update the resident fallback size in both the prompt path and the server's request splitter. The helper observes the resolved auto ceiling and the existing row-size override. Streaming, the explicit decode-window path, explicit numeric prefill, the one-shot generator and the non-Metal paths retain their previous configuration. ## Validation **Passed.** Built current main `3142b66b` and candidate `6bec84b4` with `cmake -S engine/metal -B build-metal -G Ninja -DCMAKE_BUILD_TYPE=Release -DSTRATA_GEOMETRY=flash_next -DMETAL_CPP_DIR=<metal-cpp>` and `cmake --build build-metal --target strata`. The submitted revision adds a documentation-only change to that candidate. M5 Ultra, macOS 27.2.1, 96 GB, resident AtomicChat Qwen3.8-Flash-Next AD-4.27, INT8 KV, Q2_0 MTP (`--spec 3 --spec-min-p 0.5`), context 262,144, two slots, `--expert-cache 64 --mmap-experts --conversation-cache-mib 1024`. Used an exclusive machine and a fresh engine per arm. One 4,880-token warm-up preceded each arm; the manual arms used `--prefill 4096` and `STRATA_METAL_PROMPT_CHUNK=1024`. Identical, cache-miss fixtures made from Strata documentation excerpts with a synthetic marker and temperature 0 were sent serially. Decode is a separate 47-token code prompt with a 400-token cap. | Run order | Prefill 25,523 tok, tok/s | Prefill 101,295 tok, tok/s | Decode tok/s | Peak wired increment MiB | Peak RSS GiB | | --- | ---: | ---: | ---: | ---: | ---: | | main auto | 2,122.4 | 2,062.0 | 156.8 | 70,527 | 69.98 | | main manual | 2,309.0 | 2,247.0 | 160.8 | 70,387 | 63.69 | | PR auto | 2,317.3 | 2,249.5 | 161.1 | 70,460 | 62.44 | | PR auto | 2,318.3 | 2,245.1 | 160.9 | 70,467 | 62.41 | | main manual | 2,318.3 | 2,245.1 | 160.9 | 70,470 | 64.34 | | main auto | 2,124.3 | 2,061.1 | 160.4 | 70,464 | 62.38 | All six arms returned byte-identical answers, including a 29,678-token marker check and the short code request. The new engine log showed `4096-token calls of 1024-row resident prompt chunks`. The bundled `strata bench run --url <local-server> --track custom` standard-v1 draft suite also completed on main and the submitted revision `d5fde61a`. Both finished all 21 required requests with 12/12 sanity checks. Median prompt rates were 2,448.1 → 2,421.8 tok/s for its 4K fixture and 2,460.9 → 2,456.5 for 16K. The standard fixtures did not show the speedup observed on the longer, targeted fixtures. The runner recorded the engine revision from the source checkout's `terminal/COMMIT`. **Failed during development.** An early port checked the resident mode before the Metal window existed and fell back to the original size. That run was excluded from the table. The updated engine selects it after the window is ready. **Still to validate.** Other Apple Silicon models, other packs, streaming/window fallback execution, and CUDA/HIP/SYCL builds. The affected shared source sites are Metal-guarded. This is one machine and one pack; I would appreciate a same-day comparison on another Mac before requesting review.
Sur le site
Liens install, modèles, releases.