#6434 [Engine] Use eight-row tails in M1 resident expert GEMMs
open · @Maryarty · 0 comentários · No GitHub
BenchmarksSetup & installModels & quants
Descrição
## Problem and result
On the M1 resident v2 MoE prefill path, every expert is padded to a multiple of 16 rows. A final tile with 1–8 routed rows can use an 8-row GEMM instead. On an upstream-based build, a 300-row Q2_0/Q2_0 fixture with 512 experts reduced padded rows from **8192 to 4592**, and the measured expert-chain GPU time fell from **18.155 to 15.266 ms per layer** (−15.9%).
## Approach
The opt-in `STRATA_METAL_PROMPT_M8_TAIL=1` specializes the resident v2 scan, gate/up, and down kernels on Apple M1. The scan marks the final tile and packs fewer padding rows. Gate/up and down use eight rows for marked tiles. Full tiles still use 16 rows. The tile count and allocated workspace capacity stay the same. The option defaults to off.
## Validation
M1 Max, 32 GPU cores, 64 GiB, macOS 26.6.2 (25G83). The patch is based on upstream `3142b66` and was tested at `0689549`.
**Passed.** Release Metal build of `strata`, `moe_mm_parity` and `runtime_hello`. `ctest --test-dir engine/build-metal -R '^runtime_hello$' --output-on-failure` passed. The following tests pass in both modes and produce the same GPU output hash within each pair.
```sh
for mode in v2 m8-tail; do
engine/build-metal/moe_mm_parity --"$mode" --gu Q2_0 --dn Q2_0 --T 89
engine/build-metal/moe_mm_parity --"$mode" --gu Q2_0 --dn Q5_0 --T 89
engine/build-metal/moe_mm_parity --"$mode" --gu Q4_K --dn Q5_K --embd 3072 --ff 512 --T 89
engine/build-metal/moe_mm_parity --"$mode" --bench --T 300
done
```
On my earlier local branch (`3df3e0e`), two runs per setting on the resident Qwen3.8-Flash-Next GSQ-RCO Q2_0 pack with PLE and a 256-token prefill chunk gave **239.41 → 254.85 prefill tok/s** on 2K input and **235.26 → 249.88 tok/s** on 4K. First-logit hashes and 128 generated IDs matched within each input. These full-model figures are from the earlier local branch.
**Failed or incomplete.** The first Q4_K/Q5_K fixture used the default expert width and crashed in host fixture construction with the flag off. The block-aligned shape in the command above passed with the flag off and on. The new full-model upstream run at chunk 256 was stopped by the 15% memory guard in stock and both patched modes, with zero new swapouts. The chunk-128 retry stopped during the stock load after 48 new system swapouts. The latter series was ended before testing the patched build.
**Not run.** Full-model parity and speed on this upstream revision remain unverified. Real-model runs on other packs and GPU generations remain open. I am keeping this in draft for those checks.
### Completed full-model A/B with a shared memory baseline
I enabled both resident weight releases from #6440 in every arm. The control is `0d91f17` from upstream `3142b66`, and the comparison build `a49fa7b` includes the M8 engine change and disables the scratch reuse flag. Every arm used the same Q2_0 and PLE pack, 2048 fixed input IDs, context 2304, prefill chunk 256, MTP width 2 and 128 greedy tokens. All five arms completed with identical first-logit hashes, 128 output IDs and MTP counts, and zero new swapouts. Minimum free memory was 17–18%.
The stock and first M8-off arms paid 13.22 and 16.60 s of cold prompt library setup. With setup warm, the M8-off arm took 8.61 s TTFT (239.66 prefill tok/s), and the two M8-on arms took 8.07 and 8.06 s (255.27 and 255.60 tok/s). This is one warm off measurement and two warm on measurements on the shared-memory baseline. Decode was 47.3–47.6 tok/s across all five arms. The earlier reported guarded attempts concern the exact PR binaries without the shared weight releases.
No site
Links install, modelos, releases.