#6608 [Engine] inkling2 Metal: ink_moe_gu_situ fails to compile on M1 Max, MTLCompilerService aborts with "cannot select

open · @darknaut · 7 commentaires · Sur GitHub

Setup & installServer & APIAMD / HIPModels & quants

Description

### Environment

- Strata app 0.2.19
- Engine `0.1.65-macos-arm64-metal` (`build_id: 26350e4387b8+metal`, `kernels: embedded`, `macos_min: 15.0`), shape `inkling2`
- MacBookPro18,2 — Apple M1 Max (64 GB), macOS 27.2.1 (26B131)
- Model: `Inkling-2-Mini-SQ-10.strata` (9.75 GiB) + vision pack, launched as `--max-context 131072 --draft dspark`

### Symptom

Loading the model in the app fails immediately:

```
server_exited (exit code 1): serve.server_parts.engines.EngineDied:
the engine exited before it was ready
```

Engine log, ~2-3 s after start and before any inference:

```
[strata] engine started: .../0.1.65-macos-arm64-metal/strata-inkling --serve
  --native .../Inkling-2-Mini-SQ-10.strata --max-context 131072
  --vision .../Inkling-2-Mini-Vision.strata --draft dspark
mtl: GPU keep-alive every 250 ms for 120 s after the last command buffer
[ink_metal] loaded .../Inkling-2-Mini-SQ-10.strata (build=SQ-10, minor=1, 9.75 GiB,
  trunk=646, dspark=54, experts=3072, vram_order=3072) in 0.02s
  (workSet=57.6 GiB, maxBuf=38.9 GiB)
mtl: pipeline ink_moe_gu_situ: Compilation failed due to an interrupted connection:
  XPC_ERROR_CONNECTION_INTERRUPTED. This error occurred after multiple retries.
```

Engine exit code **3**. The same failure happens for both `SQ-10` and `SQ-15`, with and
without `--vision`.

### Cause

The model itself loads fine in 0.02 s — the failure is inside Apple's shader compiler.
`MTLCompilerService` aborts (`EXC_CRASH / SIGABRT`, `termination.namespace: METAL`) while
compiling `ink_moe_gu_situ`, which kills the XPC connection and the engine dies with it:

```
"cannot select: %5002:gpr(s64) = 66 %5001:gpr, %11878:gpr"
  __abort_with_payload
  abort_with_reason
  fatalErrorHandler(void*, char const*, bool)
  llvm::report_fatal_error(llvm::Twine const&, bool)
```

IR context around the failing node — `s16` / `i32` element ops feeding a 64-bit `select`:

```
%5002:gpr(s64) = 66 %5001:gpr, %11878:gpr
%5001:gpr(s64) = 126 %5000:gpr(s32)
%11878:gpr(s64) = 120 i64 64
%5000:gpr(s32) = exact 128 %4958:gpr32, %13:gpr(s32)
%4958:gpr32(s32) = 45 %4957:gpr, %4955:gpr
%13:gpr(s32) = 120 i32 3
%4957:gpr32(s32) = 53 %4946:gpr32(s32), %10843:gpr(s16), 3
%4955:gpr(s32) = nuw nsw 127 %10934:gpr, %10935:gpr(s32)
%4946:gpr32(s32) = 69 %79:gpr32(s32), %bb.29, %5186:gpr32(s32), %bb.30
%10843:gpr(s16) = 120 i16 3
%8439:gpr(s32) = 52 %4946:gpr32(s32), %10843:gpr(s16), 2
%10059:gpr(s32) = 118 %10058:gpr(s16)
%12109:gpr(s32) = 120 i32 31
%10058:gpr(s16) = 320 25
```

Looks like the tile-metadata / `ushort`-indexed shuffle path of the MoE gate-up kernel:
`simd_shuffle_xor(v, ushort(N))` combined with the packed `uint2` tile entries
(`uint2((e << 16) | uint(rem), ...)`) and the `meta[]` block-size constants (`144u / 72u /
36u`). The Apple7 (M1 family) backend appears to have no selection pattern for the
resulting `select` on `s64`.

There are 11 `MTLCompilerService-*.ips` crash reports on this machine, one per retry, all
with the identical reason string.

### Reproduce

```sh
strata-inkling --selfcheck --native Inkling-2-Mini-SQ-10.strata
```

→ same `mtl: pipeline ink_moe_gu_situ: ...` line, exit code 3.

Also fails for `SQ-15`; fails with and without `--vision`; unchanged with
`STRATA_METAL_MATH=0 / 1 / 2`.

### What it is *not*

- **Not RAM / wired-limit / disk.** The model maps in 0.02 s and the crash is pre-inference.
- **Not a corrupt download.** Hash-valid model, fresh install right after the 0.1.64 → 0.1.65 update.
- **Not this machine's Metal in general.** On the same GPU the GGUF path
  (`components/engine/0.1.65-macos-arm64-metal/metal/engine-msl`, 58.5 GB wrapped work set,
  DFlash2 drafter, 131200 cells) loads and serves normally with 19.8 GB VRAM free. So the
  trigger is specific to the embedded `inkling2` kernel set on this GPU generation /
  macOS build.

### Workaround attempts

`STRATA_METAL_KERNELS` could point at a patched copy of the sources, and
`STRATA_METAL_MSL_DUMP` dumps the exact units being compiled — so bisecting the offending
construct is possible. I did not want to ship a hand-patched MoE kernel as a workaround
(silently wrong gate/up output is worse than a clean failure). Happy to test a candidate
build if you can produce one — e.g. avoiding 64-bit `select` in that path, or a separate
kernel variant gated on GPU family.

Sur le site

Liens install, modèles, releases.