#4272 [Engine] Metal: opt-in routed-expert tiles (STRATA_METAL_MOE_GU/_DN); Q2_0 8,1,2 + 4,2,4 is 17-24% faster on M1 Max

open · @joseedson18jc · 0 comentarios · En GitHub

BenchmarksServer & APIAMD / HIPNVIDIA / CUDAModels & quantsDocumentationWindows

Descripción

## Problem and result

On the M1/M2 family the routed experts of a **Q2_0** pack run with the gate/up tile 32,1,4 and the down tile 8,1,4 (the Q2_0 down is `moe_down_142`). On an M1 Max these are not the fastest tiles that `moe.metal` already instantiates. `expert_parity --bench` (Q2_0 gate/up + down, synthetic, M1 Max 64 GB):

| tokens | 32,1,4 + 8,1,4 (default) | 8,1,2 + 4,2,4 | |
|---|---|---|---|
| 1 | 114 us a layer | 94 | -17% |
| 2 | 214 | 167 | -22% |
| 3 | 318 | 242 | -24% |

Live (Qwen3.8-Flash-Next Q2_0, `--spec 2` + MTP, through `serve/server.py`) this is +5-7% greedy decode. This PR adds the tiles as an **opt-in knob** and does not change any default.

## Related discussion

#4238 does the same for the K quants on M1/M2 (gate/up 8,1,2) and leaves Q2_0 on the M5 tile. This is the Q2_0 counterpart, as an env knob rather than a default, because it was measured on one machine.

## Approach

`STRATA_METAL_MOE_GU` / `STRATA_METAL_MOE_DN` ("L,R,S") in `metal/src/engine/window.cpp`, read once:
- Both expert windows read it: the decode `moe` block and `mtp_encode`.
- `gx_gu` / `gx_dn` and the threadgroup widths follow the chosen tile, since they are computed from the same variables.
- Only tiles in `GU_SET` / `DN_SET` (the ones `moe.metal` instantiates) are accepted. Anything else is refused with a log line, and the default stays, so a typo cannot fail pipeline creation.
- The chosen tile is logged once (`strata metal: STRATA_METAL_MOE_GU=8,1,2`).
- Unset, the code path and the tiles are unchanged.

## Validation

**Passed:**
- `expert_parity --gu Q2_0 --dn Q2_0 --ntok {1,2,3} --gucfg 8,1,2 --dncfg 4,2,4`: y rel L2 5.3-6.3e-8, PASS (8,1,4 + 4,2,4 also passes).
- `expert_parity --bench` per config, two runs each, table above. `--sweep` picks 8,1,2 / 8,1,4 for gate/up and 4,2,4 for down at every token count.
- The parity suite on this change (built on 0.1.51 + the #3301 port, M1 Max, macOS 27): 16/17. `fuse_parity` fails on `router_route`, identically without the change (the known M1 Max routing issue).
- Live A/B, bracketed B C B C, the same binary with the knob off/on. Six unseen prompts, 450 tokens each, speed from the server's `/metrics`, no other traffic:
  - greedy: off 32.7 / 34.0, on 35.8 / 35.7 tok/s.
  - sampled (#3301 sampler): off 31.1 / 32.4, on 33.8 / 33.3 tok/s.
  - The worst "on" arm beats the best "off" arm in both modes.
- This branch (`main` + the commit) builds `strata` and `expert_parity` with no new warnings.

**Failed:**
- none observed

**Not run:**
- An engine run of this exact branch: the live A/B binary was 0.1.51 + the #3301 sampler port, and this commit applies to `main` unchanged.
- Any machine other than the M1 Max, or a pack other than Q2_0 (the K quants are #4238's domain).
- CUDA / HIP / SYCL: not touched.

## Limits

- Measured on one M1 Max 64 GB with one Q2_0 pack. M2, M3 and M4 are unmeasured, so this stays opt-in rather than an M1/M2 default.
- If a maintainer wants it as a default, the natural gate is the same M1/M2 check as #4238's.

En el sitio

Enlaces a install, modelos, releases.