#6411 [Engine] Scope Metal host-row prefetch across calls in one prompt
open · @ain-soph · 0 comentarios · En GitHub
Descripción
### Area [Engine] ### What do you want to do? Could we scope an opt-in follow-up for Metal prefill that overlaps CPU embedding and PLE row preparation across consecutive prompt calls? #6182 and #6183 cover overlap between chunks inside one call. When a single request is split into several calls for progress and cancellation, the GPU still waits for the first host rows of each new call. The fork prototype passes a hint for the next piece of the same request. The async task owns the token IDs it reads. The prototype checks the position, row count, first and last token IDs and preceding PLE tokens. For the upstream version the hint must belong to the same request and match the exact continuation; cancellation, restore and a switch to a different request must invalidate staged work. The proposed scope is the resident prompt path, default off. On one M5 Ultra (96 GB, resident AtomicChat Flash-Next AD-4.27, INT8 KV, Q2_0 MTP, two 262,144-token slots), the optimized fork ran 26,106 / 115,856-token prompts at 2,690 / 2,693 tok/s. Disabling only cross-call staging in that stack measured 2,582 / 2,575 tok/s. These are fork ablations, not measurements against current main. Before a PR I'd retest with identical requests on current main and include cancellation, checkpoint and interleaved-slot checks. Would this be useful as a separate opt-in patch? ### What do you do today instead? Use four 1,024-row chunks per call as in #6182, or set a fixed 4,096-token call size on the resident path. There is still host work at the call boundary.
En el sitio
Enlaces a install, modelos, releases.