#3284 [Engine] Batch slots on Metal, so one Mac Studio can serve a small office

说明

### Area

[Engine]  (speed, kernels, models, the server and API)

### What do you want to do?

We help small and mid-size companies, roughly 10 to 100 people, run their AI tools on hardware they own. Strata with Flash-Next is the most interesting option we've found for that. Being able to run a model this size on a Mac at all matters a lot to companies that don't want a server room full of GPUs. And thanks for the Mac port, by the way. Going from "out of scope" to a signed .dmg in under two weeks was impressive to watch.

The one thing that holds us back is that the Metal engine handles one request at a time (docs/MAC.md, Limits). For one person that's fine. For an office, it decides how many Macs you have to buy.

Here's how it looks on paper for a 100-person company. We assume about 5 model calls per person in the busiest hour, prompts of around 6k tokens (some 24k when they pull from the knowledge base) and about 700 output tokens. Using your M5 Max numbers and a rough scale-up for M5 Ultra, one request keeps a Mac busy for 9 to 12 seconds. Put that through a simple queueing model (under 3 s of waiting on average, under 10 s for 95% of requests) and you need 5 to 6 Mac Studios, spare included. Once agents are in the mix it's 8 or 9. If each Mac could decode 2 to 4 conversations together, the way the CUDA batch slots do, our rough estimate is 30 to 45% fewer machines. For a company that size, that's the difference between "a bit expensive" and an easy yes.

So we wanted to ask whether batch slots on Metal (`"parallel": N`, opt-in, same as on CUDA/HIP) are something you'd consider.

From #2715 the gap looks known: `--batch` is accepted on Metal, and the engine exits with code 86 on the first window that has two sequences. That report lists 17 entry points without a Metal implementation yet. It was closed in the 0.2.0 reset and nobody has picked it up since. We haven't re-tested it on 0.2.0-beta ourselves (we don't have a Mac with enough memory yet), so we're not asking to reopen it as a bug.

A few questions:

1. Is Metal batching on your list for 0.2.x, or should Mac setups plan on one request per machine for now?
2. Until it exists, would it make sense to refuse `--batch` on Metal at startup with a clear message, instead of exiting on the first concurrent request?
3. If you'd accept outside help, what order would you want (the capability queries and mapped copies first, then the multi-row kernels?), and what parity evidence per kernel?

We'd be glad to help. When our Mac Studios arrive (M5 Ultra, 256 GB, UD-Q6_K_XL) we can run whatever you'd like measured and post it as [Benchmark] reports. If you agree on the scope, we're also open to porting some of the kernels ourselves in small PRs.

### What do you do today instead?

Plan for one request per Mac and put several Macs behind a router with sticky sessions, along the lines of strata-router (#772). It works, but it's the expensive way to get concurrency.

The JSC UNITE GROUP team