#6495 [Engine] Layer-major streaming prefill on Metal

open · @Maryarty · 0 comentários · No GitHub

BenchmarksServer & APIModels & quants

Descrição

### Area

[Engine] (speed, kernels, models, the server and API)

### What do you want to do?

Reduce the time to first token on long fresh prompts when Strata streams experts on Apple Silicon. On my M1 Max with 64 GiB, an opt-in layer-major Metal prototype reduced this wait by 35 seconds at 16K and 44 seconds at 32K.

The current Metal streaming path completes every layer for one prompt chunk, then revisits the experts for the next chunk. The prototype runs a bounded group of full chunks through each layer while its expert weights are available and prefetches the next layer once per group. GDN and attention state advance in token order within each layer. The group publishes progress and a checkpoint only once the entire group is committed. Partial tails fall back to the existing path.

### What do you do today instead?

Use the default chunk-major streaming path with a 12 GB expert arena. This keeps the model within the memory budget, but processing a long prompt can reread hundreds of GB of expert payload before the first token.

### Local measurements

Apple M1 Max (32 GPU cores, 64 GiB unified memory), Qwen3.8-Flash-Next GSQ-RCO Q2_0 with PLE, 12 GB expert arena, 1,024-token chunks, 128 greedy output tokens, MTP disabled in both arms. The only A/B change was enabling groups of up to four full chunks. One pair per prompt length on a local experimental branch.

| Fresh input | Prefill, tok/s | TTFT, s | Expert file payload, GB | Peak process footprint, GB |
| --- | ---: | ---: | ---: | ---: |
| 16,384 tokens | 160.01 → 242.91 | 102.48 → 67.65 | 327.95 → 187.73 | 17.98 → 18.13 |
| 32,768 tokens | 199.18 → 271.09 | 164.62 → 121.02 | 665.38 → 353.95 | 18.30 → 18.39 |

First-token logits matched bit for bit and all 128 generated token IDs matched in both pairs. All four runs had zero new macOS swapouts. The payload counter sums completed expert file reads in batched prefill, including prefetch. It excludes arena fill and decode. Four 1,024-token chunks needed about 210 MB of extra Metal scratch.

### MTP, warm continuation, and STOP

A separate correctness check used 512-token chunks with MTP T2 enabled. In the grouped arm I sent STOP at the first progress event on a 10,240-token prefill. The engine finished the in-flight group, returned `DONE 0 … cancel` at 4,096 processed tokens, and was ready to serve again. Cancellation took 11.1 seconds from sending STOP in this run. The next 4,096-token request started fresh, and a linked follow-up reused 4,223 cached tokens in both arms. Compared with chunk-major, both responses had bit-identical first-token logits, identical 128 generated IDs per response, and identical MTP acceptance counts (51/76 and 59/68). No new swapouts. This run checks correctness after interruption and reuse. The two arms did different work before the first request, so their decode speeds need a separate A/B.

### Proposed scope

Would a focused, opt-in Metal streaming text PR fit? I'd propose keeping the existing default, bounding group size by the available host RAM and Metal working set, and keeping batch, DFlash and vision on the existing paths. The tradeoffs are extra scratch and coarser progress and STOP boundaries, so a conservative chunk cap is important. I'd include reproduction notes and the per-run data. The scheduling idea was prompted by [strata-mlx's layer loop](https://github.com/yibie/strata-mlx/blob/main/strata_mlx/bylayer.py).

No site

Links install, modelos, releases.