#6253 [Engine] macOS / Metal: open contributions, rebased on v0.2.16

closed · @mnott · 3 comentarios · En GitHub

BenchmarksServer & APINVIDIA / CUDAModels & quants

Descripción

### Area

[Engine]  (speed, kernels, models, the server and API)

### What do you want to do?

Run the Gyro models (M, L) on a Mac with Metal on v0.2.16. Measured on an Apple M5 Max with 128 GB of unified memory. Everything from the v0.2.13 overview (#6028) landed upstream in v0.2.16, thank you:

- #6024 (Metal wraps a Gyro file without its PLE table): on v0.2.16 Gyro-M TQ2_0 wraps 37.57 GB as 2 separate ranges of 91.97 GB; greedy output identical across 5 runs.
- #6025 (Metal kernels for the pair-trellis expert types TQP5/6/7 used by Gyro-L): `tqp_parity` passes on the real Gyro-L file (144 tensors) and `expert_parity` passes for TQP5/6/7 and TQ2_T.
- #6026 (a published PLE-only GGUF passed to `--ple-gguf` is not read as a model shard).
- #6032 (serve tool-call recovery): taken upstream as narrow repairs; the remaining ideas were declined and are not re-proposed.
- #6027 (VisionArgs serve test): passes on v0.2.16 on macOS without the patch, so it is closed rather than rebased.

Open from our side: nothing. A live serving check of Gyro-L on v0.2.16 is still to come; if it shows a problem it will be reported as its own issue with numbers.

One observation to check, not a request yet: on Gyro-M (CLI, greedy, same prompt, warm page cache) decode measured 50 and 68 tok/s on v0.2.16 against 65 and 79 tok/s on our v0.2.13 build. Only a few runs; it will be re-measured on Gyro-L before anything is filed.

### What do you do today instead?

Gyro-M runs on v0.2.16 as is; Gyro-L still serves from the v0.2.13 build until the live check.

En el sitio

Enlaces a install, modelos, releases.