#6608 [Engine] inkling2 Metal: ink_moe_gu_situ fails to compile on M1 Max, MTLCompilerService aborts with "cannot select
open · @darknaut · 7 comentarios · En GitHub
Setup & installServer & APIAMD / HIPModels & quants
Descripción
### Environment - Strata app 0.2.19 - Engine `0.1.65-macos-arm64-metal` (`build_id: 26350e4387b8+metal`, `kernels: embedded`, `macos_min: 15.0`), shape `inkling2` - MacBookPro18,2 — Apple M1 Max (64 GB), macOS 27.2.1 (26B131) - Model: `Inkling-2-Mini-SQ-10.strata` (9.75 GiB) + vision pack, launched as `--max-context 131072 --draft dspark` ### Symptom Loading the model in the app fails immediately: ``` server_exited (exit code 1): serve.server_parts.engines.EngineDied: the engine exited before it was ready ``` Engine log, ~2-3 s after start and before any inference: ``` [strata] engine started: .../0.1.65-macos-arm64-metal/strata-inkling --serve --native .../Inkling-2-Mini-SQ-10.strata --max-context 131072 --vision .../Inkling-2-Mini-Vision.strata --draft dspark mtl: GPU keep-alive every 250 ms for 120 s after the last command buffer [ink_metal] loaded .../Inkling-2-Mini-SQ-10.strata (build=SQ-10, minor=1, 9.75 GiB, trunk=646, dspark=54, experts=3072, vram_order=3072) in 0.02s (workSet=57.6 GiB, maxBuf=38.9 GiB) mtl: pipeline ink_moe_gu_situ: Compilation failed due to an interrupted connection: XPC_ERROR_CONNECTION_INTERRUPTED. This error occurred after multiple retries. ``` Engine exit code **3**. The same failure happens for both `SQ-10` and `SQ-15`, with and without `--vision`. ### Cause The model itself loads fine in 0.02 s — the failure is inside Apple's shader compiler. `MTLCompilerService` aborts (`EXC_CRASH / SIGABRT`, `termination.namespace: METAL`) while compiling `ink_moe_gu_situ`, which kills the XPC connection and the engine dies with it: ``` "cannot select: %5002:gpr(s64) = 66 %5001:gpr, %11878:gpr" __abort_with_payload abort_with_reason fatalErrorHandler(void*, char const*, bool) llvm::report_fatal_error(llvm::Twine const&, bool) ``` IR context around the failing node — `s16` / `i32` element ops feeding a 64-bit `select`: ``` %5002:gpr(s64) = 66 %5001:gpr, %11878:gpr %5001:gpr(s64) = 126 %5000:gpr(s32) %11878:gpr(s64) = 120 i64 64 %5000:gpr(s32) = exact 128 %4958:gpr32, %13:gpr(s32) %4958:gpr32(s32) = 45 %4957:gpr, %4955:gpr %13:gpr(s32) = 120 i32 3 %4957:gpr32(s32) = 53 %4946:gpr32(s32), %10843:gpr(s16), 3 %4955:gpr(s32) = nuw nsw 127 %10934:gpr, %10935:gpr(s32) %4946:gpr32(s32) = 69 %79:gpr32(s32), %bb.29, %5186:gpr32(s32), %bb.30 %10843:gpr(s16) = 120 i16 3 %8439:gpr(s32) = 52 %4946:gpr32(s32), %10843:gpr(s16), 2 %10059:gpr(s32) = 118 %10058:gpr(s16) %12109:gpr(s32) = 120 i32 31 %10058:gpr(s16) = 320 25 ``` Looks like the tile-metadata / `ushort`-indexed shuffle path of the MoE gate-up kernel: `simd_shuffle_xor(v, ushort(N))` combined with the packed `uint2` tile entries (`uint2((e << 16) | uint(rem), ...)`) and the `meta[]` block-size constants (`144u / 72u / 36u`). The Apple7 (M1 family) backend appears to have no selection pattern for the resulting `select` on `s64`. There are 11 `MTLCompilerService-*.ips` crash reports on this machine, one per retry, all with the identical reason string. ### Reproduce ```sh strata-inkling --selfcheck --native Inkling-2-Mini-SQ-10.strata ``` → same `mtl: pipeline ink_moe_gu_situ: ...` line, exit code 3. Also fails for `SQ-15`; fails with and without `--vision`; unchanged with `STRATA_METAL_MATH=0 / 1 / 2`. ### What it is *not* - **Not RAM / wired-limit / disk.** The model maps in 0.02 s and the crash is pre-inference. - **Not a corrupt download.** Hash-valid model, fresh install right after the 0.1.64 → 0.1.65 update. - **Not this machine's Metal in general.** On the same GPU the GGUF path (`components/engine/0.1.65-macos-arm64-metal/metal/engine-msl`, 58.5 GB wrapped work set, DFlash2 drafter, 131200 cells) loads and serves normally with 19.8 GB VRAM free. So the trigger is specific to the embedded `inkling2` kernel set on this GPU generation / macOS build. ### Workaround attempts `STRATA_METAL_KERNELS` could point at a patched copy of the sources, and `STRATA_METAL_MSL_DUMP` dumps the exact units being compiled — so bisecting the offending construct is possible. I did not want to ship a hand-patched MoE kernel as a workaround (silently wrong gate/up output is worse than a clean failure). Happy to test a candidate build if you can produce one — e.g. avoiding 64-bit `select` in that path, or a separate kernel variant gated on GPU family.
En el sitio
Enlaces a install, modelos, releases.