Issues / #2715
#2715 `--batch` on the Metal backend is accepted at boot and exits the engine with code 86 on the first request that runs two sequences together
open · @alvarorsouza-arch · 0 Kommentare · Auf GitHub
BenchmarksSetup & installServer & APINVIDIA / CUDAModels & quantsDocumentationWindows
Beschreibung
## What happened `--batch` is accepted on the Metal backend, allocates its slot sessions at boot, and then stops the engine the first time a request actually decodes two sequences together. On a Mac that is worse than a refusal, because the server restarts the engine between requests and the rest of the run fails the same way. What I ran: setup's own config for the IQ3_S pack plus `--batch N` in `--serve`, then the repo's burst client. ```bash python3 tools/serve_load.py http://127.0.0.1:8012 --clients 1,4 --rounds 3 --max-tokens 256 ``` The boot is happy, and it announces itself with the CUDA device name on a Mac build: ```text strata generate: --batch: 4 slot sessions on CUDA0 (1.73 GiB each); 95.16 GiB free strata generate: --batch 4: the slot sessions take 6.91 GiB of VRAM on CUDA0 that the expert cache would otherwise hold strata verify: batch windows of up to 4 sequences (layers [0, -1)) ``` The first concurrent round then writes two lines and the process is gone: ```text strata batch: slot 0 takes 67 tokens (copied in 12.7 ms) strata metal: strata::kernels::pdl_supported has no Metal implementation yet ``` `metal/src/engine/missing.cpp` documents what follows that message, verbatim: `Default: say which one and stop (exit code 86), so a run never continues on outputs nobody wrote.` The server's side: ```text [strata] the engine stopped unexpectedly (exit code 86). ... The next request starts the engine again. [strata] the engine had stopped (exit code 86); starting it again (a minute or two) ... ``` In my engine log the restarts after these exits are 6, 25 and 74 s apart. Same at `--batch 2` and `--batch 3`. Measured with `tools/serve_load.py`, three rounds, 256 tokens out, one engine boot per row: | `--batch` | one client, total tok/s | that many clients, total tok/s | first token p50/p95, seconds | |---:|---:|---:|---| | 2 | 43.3 | 0.4 | 5.12 / 5.21 | | 3 | 43.3 | 0.3 | 5.19 / 5.23 | | 4 | 44.2 | 0.4 | 5.26 / 5.29 | One client is normal decode; from two sequences onward the number is the restart cycle, not throughput. ## Strata version, GPU, OS 0.1.44.1-metal, source build of `main` at `296b59d` · Apple M3 Max, 40-core GPU, 128 GB unified memory · macOS 26.6, high power mode. Model: original Flash-Next IQ3_S, `--max-context 65536`, experts resident with `--mmap-experts`, `--spec 2` with the MTP draft layer, everything else at setup defaults. ## Engine log The two relevant lines are quoted above, and the version line in the same log reads `strata generate: session is up (engine 0.1.44.1-metal)`. The full measurement set for this machine is in #2649. ## Why it stops at that call `src/core/verify.cpp:901` decides the batch window with `pdl_window(G == 1 && pdl_supported())`. `pdl_supported()` is a capability query rather than a kernel, and on Metal it is one of the entry points listed in `metal/src/engine/stubs.cpp`, so it runs `missing()` and exits. ## How far the batch route is from working on Metal `STRATA_METAL_STUBS=log`, which that same file describes as `for listing which entry points a code path reaches, never for answers`, shows what a four sequence run touches. Seventeen entry points: ```text pdl_supported, pdl_scope fused_gr_read_multi, gdn_conv_l2_multi, gdn_ab_multi, gdn_step_norm_multi, shared_expert_multi, broadcast_streams native_quantize_q8_1, native_mmvq, native_router_top10, quantize_q8_1_rows, iq_embed_rows, bf16_gemv_fp32_mmvf doorbell_publish_res, copy_from_mapped, copy_i32_from_mapped ``` Two capability queries, the multi row family a batched window needs, the native helpers, and the expert pool side, which Metal does not use because it reads experts in place from the mapped file. ## What I would suggest - Refuse `--batch` on Metal at boot, the way `--kv int8` already stops there with `strata metal: strata::kernels::kv_block_bytes has no Metal implementation yet`, instead of allocating 1.73 GiB per sequence and exiting 86 mid-request. - If the intent is to reach batch on Metal, `pdl_supported` returning false in the stubs moves the stop from a capability query to the kernels that matter, which is where the list above becomes a checklist. - `docs/MAC.md` still names `--parallel` (batch slots) as not ported. The flag today is `--batch` (and `--slots`), so a Mac reader can miss the line, and the boot line's `on CUDA0` does not help.
Mehr auf der Site
Links zu Install, Modellen, Releases.