#3756 [Engine] Metal: the control vector (experimental speed projection) on the Mac engine

open · @khaterdev · 0 comments · View on GitHub

BenchmarksSetup & installServer & APIAMD / HIPNVIDIA / CUDAModels & quantsSecurityDocumentationWindows

Description

## Problem and result

On a Mac the experimental speed projection (the control vector in `data/experimental-speed-projection`, `--control-vector-scaled`) did nothing: `metal/src/engine/missing.cpp` returned an empty vector and `docs/MAC.md` listed it as not ported, while the CUDA, HIP and SYCL engines apply it. A Mac user who turned it on got the stock model.

After this change the Metal engine applies the vector where the PC engines do: after each steered layer's FFN write, in the decode window and in both prompt paths. The per-request switch (`"experimental_speed_projection": false` in a request, the web app's drawer) works as on a PC, and a request with it off decodes exactly what the stock engine decodes. `./setup.sh` on a Mac asks the same question as the PC installers (or takes `--experimental-speed-projection on|off|GGUF`); the app's Mac setup takes the flag; `strata launch` keeps it in its Mac argument list.

Off by default, as everywhere else. Nothing changes for a Mac that does not load a vector: `cv_covers()` is false, so no dispatch is added and every read keeps its `apply` flag.

## Related discussion

None. Opened directly; if the scope should go through an issue first, say so and I will move the discussion there.

## Approach

- `metal/src/engine/msl/cvec.metal`: one kernel per (stream, row), `cvec.cu`'s math: the layer's pending FFN write with `gr_norm`'s arithmetic (`h += bo * 2 sigmoid(inj / hc)`), then `h -= s (h . v) v` (project) or `h += v` (add). With the request flag 0 or no direction for the layer it applies only the write, so the next read (`apply = 0`) sees bitwise what the fused read would have produced.
- `window.cpp`: `pend_write(l)` decides whether layer `l`'s first read applies the pending write (not after a steered layer: the kernel did), and the final read likewise. `cv_ready()` uploads the loader's tables into window buffers once and compiles the kernel; `cv_dispatch()` is pushed after `moe_combine_sh` (decode window), `moe_out` (both prompt chains). The dispatch cache is unaffected: a vector is loaded before the first window, and the per-request switch is a flag buffer the kernel reads.
- `cvec.cpp`: the Metal side of `cvec.hpp`: `cvec_upload` keeps the host tables and `Cvec`, `cvec_set_enabled` writes the window's flag between requests (nothing is in flight on Metal then). `cvec_apply`/`cvec_tables` stay stubs: they serve the CUDA graphs and the CUDA prefill, which the Metal engine does not run. The three resolved stubs were removed from the generated `stubs.cpp` by hand (`gen_stubs.py` would drop the same lines).
- `setup.py`: the PC installers' choice became `esp_choose()` / `esp_engine_args()`, used unchanged by the PC path and now by `mac_control_setup` (flag only, the app asks nothing) and `metal/setup_mac.py` (asks, default no). `strata_cli/launch/runner.py`: the Mac filter keeps `--experimental-speed-projection` (it passes `off`, as for the PC).
- Docs: `MAC.md` (the limit removed, what was measured), `DETAILS.md` (the Mac setup line).

Compatibility: no engine flag, config key or protocol line changes. PC engines are untouched (`metal/` and the shared Python only).

## Validation

Environment: MacBook Pro M1 Max, 64 GB, macOS 26.5.2; Apple clang 21.0.0, metal-cpp_26, llama.cpp 3cf0325; Qwen3.8-Flash-Next GSQ-RCO Q2_0, 256K context, `--spec 2 --mtp`, the picture reader on. Revision f1a0905d, `cmake -S metal -B build-metal -G Ninja -DCMAKE_BUILD_TYPE=Release -DMETAL_CPP_DIR=... -DSTRATA_GGML_DIR=...; ninja -C build-metal strata cvec_parity`. The stock reference is the released engine component 0.1.47 (`76f90e58436d+metal`, the parent of this branch).

**Passed:**

- `./build-metal/cvec_parity` -> `control vector parity, 24 windows (project/add, with/without the write, 1..8 rows): worst rel err 1.18e-07, off = gr_norm's write bitwise: 0 failures  PASS`
- Engine A/B, greedy, `reasoning_effort: none`, 9 prompts (a short question, code, a 434-token and an 11,295-token prompt, five requests the model refuses), served by `serve.server` with the vector loaded (`--control-vector-scaled <esp>.gguf:1.0 --control-vector-layer-range 4 44 --cvec-mode project --cvec-dir per-layer`):
  - switched off per request: all 9 answers are the stock engine's, character for character;
  - switched on: the four ordinary prompts answer normally (two of them with a differently worded answer, as the package describes), and all five refused prompts are answered.
- Speed on the same server (`speed.py`: 300-token story decode, median of 4 runs after a warm-up; an 8.5K-token prompt read twice), alternating on/off twice: decode on 31.9 and 32.5 tok/s, off 33.3 and 33.2 tok/s (about 3% lower with the vector on); prompt read on 251-345, off 285-345 tok/s (within the run-to-run spread).
- `python3 -m unittest tools.test_setup_mac` OK; `tests.terminal.test_own_files` OK; `tests.terminal.test_launch` 107 of 108 (the Mac argument-list test updated for the kept flag).

**Failed:**

- `tests.terminal.test_launch.SetupArgs.test_the_apps_own_installer_arguments` fails before and after this change on this Mac (on 011c9e60 with nothing changed): it expects `--kv` in the app's installer arguments, which the Mac filter drops. Not touched here.

**Not run:**

- CUDA, HIP and SYCL builds: the C++ change is under `metal/` only; `setup.py`'s PC path calls the same code it did, through the new helper, but was not executed on a PC.
- The desktop app on a Mac with the flag (the app writes `user_args` itself); the Swift, Coder, Unsloth and Gyro families with a vector (the loader's family checks are the PC ones).
- Pictures with the vector on.

## Limits

- As on the PC engines, the vector follows the main model's layers only; the draft layer proposes and the main model verifies, so the output is the steered model's.
- The ~3% decode cost measured here is one kernel per steered layer per window (41 dispatches of 8 threadgroups); it could be folded into `gr_norm`'s read as the CUDA prefill does (`gr_write_cvec_norm_rs`), not done here.

Related on this site

Quick links to related pages.