问题 / #2715

#2715 `--batch` on the Metal backend is accepted at boot and exits the engine with code 86 on the first request that runs two sequences together

open · @alvarorsouza-arch · 0 评论 · 去 GitHub 看

BenchmarksSetup & installServer & APINVIDIA / CUDAModels & quantsDocumentationWindows

说明

## What happened

`--batch` is accepted on the Metal backend, allocates its slot sessions at boot, and then stops the engine the first time a request actually decodes two sequences together. On a Mac that is worse than a refusal, because the server restarts the engine between requests and the rest of the run fails the same way.

What I ran: setup's own config for the IQ3_S pack plus `--batch N` in `--serve`, then the repo's burst client.

```bash
python3 tools/serve_load.py http://127.0.0.1:8012 --clients 1,4 --rounds 3 --max-tokens 256
```

The boot is happy, and it announces itself with the CUDA device name on a Mac build:

```text
strata generate: --batch: 4 slot sessions on CUDA0 (1.73 GiB each); 95.16 GiB free
strata generate: --batch 4: the slot sessions take 6.91 GiB of VRAM on CUDA0 that the expert cache would otherwise hold
strata verify: batch windows of up to 4 sequences (layers [0, -1))
```

The first concurrent round then writes two lines and the process is gone:

```text
strata batch: slot 0 takes 67 tokens (copied in 12.7 ms)
strata metal: strata::kernels::pdl_supported has no Metal implementation yet
```

`metal/src/engine/missing.cpp` documents what follows that message, verbatim: `Default: say which one and stop (exit code 86), so a run never continues on outputs nobody wrote.` The server's side:

```text
[strata] the engine stopped unexpectedly (exit code 86). ... The next request starts the engine again.
[strata] the engine had stopped (exit code 86); starting it again (a minute or two) ...
```

In my engine log the restarts after these exits are 6, 25 and 74 s apart. Same at `--batch 2` and `--batch 3`.

Measured with `tools/serve_load.py`, three rounds, 256 tokens out, one engine boot per row:

| `--batch` | one client, total tok/s | that many clients, total tok/s | first token p50/p95, seconds |
|---:|---:|---:|---|
| 2 | 43.3 | 0.4 | 5.12 / 5.21 |
| 3 | 43.3 | 0.3 | 5.19 / 5.23 |
| 4 | 44.2 | 0.4 | 5.26 / 5.29 |

One client is normal decode; from two sequences onward the number is the restart cycle, not throughput.

## Strata version, GPU, OS

0.1.44.1-metal, source build of `main` at `296b59d` · Apple M3 Max, 40-core GPU, 128 GB unified memory · macOS 26.6, high power mode. Model: original Flash-Next IQ3_S, `--max-context 65536`, experts resident with `--mmap-experts`, `--spec 2` with the MTP draft layer, everything else at setup defaults.

## Engine log

The two relevant lines are quoted above, and the version line in the same log reads `strata generate: session is up (engine 0.1.44.1-metal)`. The full measurement set for this machine is in #2649.

## Why it stops at that call

`src/core/verify.cpp:901` decides the batch window with `pdl_window(G == 1 && pdl_supported())`. `pdl_supported()` is a capability query rather than a kernel, and on Metal it is one of the entry points listed in `metal/src/engine/stubs.cpp`, so it runs `missing()` and exits.

## How far the batch route is from working on Metal

`STRATA_METAL_STUBS=log`, which that same file describes as `for listing which entry points a code path reaches, never for answers`, shows what a four sequence run touches. Seventeen entry points:

```text
pdl_supported, pdl_scope
fused_gr_read_multi, gdn_conv_l2_multi, gdn_ab_multi, gdn_step_norm_multi, shared_expert_multi, broadcast_streams
native_quantize_q8_1, native_mmvq, native_router_top10, quantize_q8_1_rows, iq_embed_rows, bf16_gemv_fp32_mmvf
doorbell_publish_res, copy_from_mapped, copy_i32_from_mapped
```

Two capability queries, the multi row family a batched window needs, the native helpers, and the expert pool side, which Metal does not use because it reads experts in place from the mapped file.

## What I would suggest

- Refuse `--batch` on Metal at boot, the way `--kv int8` already stops there with `strata metal: strata::kernels::kv_block_bytes has no Metal implementation yet`, instead of allocating 1.73 GiB per sequence and exiting 86 mid-request.
- If the intent is to reach batch on Metal, `pdl_supported` returning false in the stubs moves the stop from a capability query to the kernels that matter, which is where the list above becomes a checklist.
- `docs/MAC.md` still names `--parallel` (batch slots) as not ported. The flag today is `--batch` (and `--slots`), so a Mac reader can miss the line, and the boot line's `on CUDA0` does not help.

本站相关内容

相关页面的快捷入口。