#4019 [Engine] Mac: calibrate the decode kernels' tiles on the device (setup.sh --calibrate)

open · @rasa04 · 0 comentários · No GitHub

BenchmarksSetup & installMulti-GPUAMD / HIPNVIDIA / CUDAModels & quantsWindows

Descrição

**Related discussion:** #4025.

**Problem.** The Metal decode kernels use tiles tuned on the M4 and M5. On an M1 Max (24-core GPU) the fp32 dense GEMVs (`STRATA_METAL_F16` unset) of a Qwen3.8-Flash-Next Q2_0 decode window take 33.1 ms on the default tiles and 20.7 ms on each shape's best one (`gemv_parity --one … --ntok 2`). `setup.sh --calibrate` is not ported to the Mac.

**Change** (opt-in; with neither variable set the dispatches are unchanged):
- `STRATA_METAL_CALIBRATE=<file>`: the engine times every instantiated tile of the dense GEMVs, `gr_down` and the routed experts on the device, outputs into a scratch buffer, and keeps a tile only if it is ≥ 3% faster.
- `STRATA_METAL_TILES=<file>`: use the tiles, never measure. A shape the file lacks, a file from another GPU, or an expert tile that does not divide the model's rows keeps the default.
- `setup.sh --calibrate` on a Mac (`tools/calibrate.py`, like the HIP `STRATA_SH_STREAM` step): measures, races the tiles against the defaults over 3 interleaved rounds, and writes `STRATA_METAL_TILES` into the config only above `MIN_GAIN`. A round whose engine fails to start or dies is dropped, as in the other steps (`measure_candidate`, 41188ec7). One file per model.
- The fp16 decode (`STRATA_METAL_F16=1`, the default of a resident Q2_0 / IQ3_S pack): the fp16 GEMV and expert kernels keep their tiles - the calibration neither measures them nor gives them a tile measured on the fp32 kernels. The fp32 dispatches of its windows (the formats without an fp16 kernel, `gr_down`, the experts off the fp16 path) are measured and tiled.
- The Q5_0 down experts (`cbef1a9f`) are among the calibrated down formats (`DN_SET(T_Q5_0)` has every candidate tile).
- With `STRATA_METAL_HCLD=1` (off by default) `gr_down` keeps its `gr_downp_*_8_8` kernels: no tile from the file, none measured.
- `setup.sh --calibrate` does not start beside a running engine; `strata engine calibrate` refuses a Mac config (its tiles would replace the file the config names).

**Result** (M1 Max, 64 GB, Q2_0 resident, MTP, before the fp16 decode existed; `tools/ab_engine.py`, 3 rounds, medians, tok/s):

| | default | tiles |
|---|---:|---:|
| decode, short chats | 25.1-28.3 | 32.6-36.8 (+30-32%) |
| decode after a 6K-token prompt | 23.2 | 29.2 (+26%) |
| short prompt read | 39.1-40.2 | 52.2-53.8 |
| 6K-token prompt read | 251.1 | 251.9 |

Quality (`tools/quant_quality.py`, 4995 positions): same top token 98.4%, mean KL 0.003, perplexity 4.439 → 4.431. Near-ties can flip, since a tile changes the order of a sum.

With the fp16 decode on (`STRATA_METAL_F16=1`; `setup.sh --calibrate`'s race, 3 interleaved rounds, medians of 6 rates, tok/s): run 1 (`7ed042b4` on `ffa80d73`) default 40.6, tiles 43.35 (+6.8%, kept); run 2 (`3c7874fb` on `ffa80d73`) default 40.6, tiles 41.7 (+2.7%, under `MIN_GAIN`: the defaults stayed); run 3 (`7580f99d` on 0.2.12, its config with `--spec 3 --spec-min-p 0.5` and a 128K context) default 35.85, tiles 36.85 (+2.8%, under `MIN_GAIN`). Quality of run 1's tiles (`quant_quality`, 4995 positions) against the fp16 default: same top token 97.7%, mean KL 0.008, perplexity 4.415 → 4.433; against fp32, mean KL 0.0079 with the tiles and 0.0091 without.

**Validation** (`6a685443` on 0.2.12 `146df3c4`, M1 Max, macOS 27):
- Passed: build; default path byte-identical to `146df3c4` with `STRATA_METAL_F16` unset and set to 1 (`quant_quality`, 4995/4995 rows each); `expert_parity --dn Q5_0` 36/36 (every down tile × ntok 1, 2, 4; on `9f7d977d`); `python -m unittest tools/test_calibrate.py tools/test_setup_mac.py tests/terminal/test_calibration.py` (108 of 109, incl. the new `test_metal_tiles`, `test_engine_death_during_a_metal_round_keeps_the_run` and `test_a_mac_config_is_refused`); `setup.sh --calibrate`'s measure and race (`calibrate.run` on the Q2_0 config, `STRATA_METAL_F16=1`) twice on `ffa80d73` and once on 0.2.12.
- Failed: `test_failed_calibration_says_the_engines_reason`, which fails on a Mac on `146df3c4` without this change too (`metal/setup_mac.py` is missing from the test's temp tree).
- Not re-run on `146df3c4`: `gemv_parity`, `gr_parity` (6 variants), `expert_parity` for the other formats (no kernel changed; 507/507 and all-pass on 0.2.7 `5e8a4cca`). The `gr_down` limit check from a tiles file: built, not run (no Mac here has a variant over the limit).
- Not run: other Macs (M2-M5, Pro / Max / Ultra), SSD streaming, `--parallel`, the IQ3 / Unsloth / Gyro packs, CUDA / HIP / SYCL (the engines not touched).

**Limits.** The gain is confirmed only on the Mac and `--spec` the calibration ran on. With the fp16 decode on, the decode gain comes from the fp32 dispatches of a decode window (the formats without an fp16 kernel) and `gr_down`; the prompt's windows are tiled too but do not count in decode tok/s. Window sizes it did not see and the MTP layer keep the defaults; with SSD streaming the experts do too. Re-run after an update that changes the kernels.

<details><summary>Individual runs (tok/s)</summary>

```text
request          tiles    decode (rounds 1/2/3)    prompt read (rounds 1/2/3)
chat 1           default  30.7 / 27.6 / 28.3       42.3 / 38.1 / 39.1
chat 1           tiles    36.8 / 36.5 / 41.8       51.4 / 52.2 / 58.6
chat 2           default  27.0 / 24.8 / 25.1       42.9 / 39.4 / 40.2
chat 2           tiles    33.2 / 32.6 / 37.0       53.6 / 53.8 / 59.6
chat 3           default  27.0 / 25.0 / 25.1       42.3 / 39.1 / 38.8
chat 3           tiles    32.6 / 32.2 / 33.7       51.3 / 52.6 / 58.6
6K-token prompt  default  23.6 / 23.2 / 22.7       268.4 / 251.1 / 248.6
6K-token prompt  tiles    29.2 / 28.8 / 30.0       251.9 / 251.6 / 253.3
```

`setup.sh --calibrate`: default 25.9 / 26.1 / 27.8, tiles 35.0 / 35.1 / 37.3.

`setup.sh --calibrate` with `STRATA_METAL_F16=1` (the report's rates): run 1 default 40.8 / 40.7 / 40.5 / 40.6 / 40.5 / 40.6, tiles 43.5 / 43.4 / 43.2 / 43.1 / 43.3 / 43.4; run 2 default 40.6 / 40.7 / 40.6 / 40.5 / 40.7 / 32.0, tiles 41.8 / 41.8 / 41.7 / 41.7 / 41.7 / 41.7; run 3 default 35.6 / 35.4 / 36.0 / 35.9 / 36.0 / 35.8, tiles 37.0 / 36.7 / 37.2 / 37.0 / 36.6 / 36.3.
</details>

<details><summary>Local changes to tools/ab_engine.py (whole answers kept, the die cooled before each start)</summary>

```diff
--- tools/ab_engine.py
+++ the local copy
@@ -22,7 +22,7 @@
 import urllib.request
 from pathlib import Path
 
-ROOT = Path(__file__).resolve().parents[1]
+ROOT = Path("<the Strata checkout>")
 PROMPTS = (
     "Write a Python function that merges two sorted lists into one sorted list, with a docstring and two tests.",
     "Explain in two paragraphs how a refrigerator moves heat from inside to outside.",
@@ -64,6 +64,19 @@
     raise TimeoutError("server not ready")
 
 
+def tdie() -> float:
+    d = json.load(open("<the die temperature file>"))
+    return next(g["avg"] for g in d["temps"]["groups"] if "die" in g["name"].lower())
+
+
+def cool(limit: float) -> None:
+    """Each arm starts on a cool chip: a hot M1 Max throttles and the arm that runs later loses."""
+    t0 = time.time()
+    while (t := tdie()) >= limit and time.time() - t0 < 900:
+        time.sleep(15)
+    print(f"tdie {t:.1f} C", flush=True)
+
+
 def main() -> int:
     ap = argparse.ArgumentParser()
     ap.add_argument("config")
@@ -83,9 +96,12 @@
     reqs = [(f"chat{i}", p) for i, p in enumerate(PROMPTS)] + [("long", long_prompt(a.long_words))]
     out = open(a.log, "a")
     res: dict = {n: {} for n, _ in arms}
+    cool_limit = max(50.0, tdie() + 1.0)   # the idle chip now, +1 C: equal starts without waiting for a cold room
+    print(f"cool limit {cool_limit:.1f} C", flush=True)
     for rnd in range(a.rounds):
         order = arms if rnd % 2 == 0 else arms[::-1]
         for name, env in order:
+            cool(cool_limit)
             cfg = dict(base)
             e = dict(env)
             if "exe" in e:
@@ -110,7 +126,7 @@
                     row = {"round": rnd, "arm": name, "req": tag, "decode": t.get("predicted_per_second"),
                            "prompt": t.get("prompt_per_second"), "prompt_n": t.get("prompt_n"),
                            "n": r.get("usage", {}).get("completion_tokens"),
-                           "text": r["choices"][0]["message"].get("content", "")[:80]}
+                           "text": r["choices"][0]["message"].get("content", "")}
                     out.write(json.dumps(row) + "\n")
                     out.flush()
                     res[name].setdefault(tag, []).append(row)
```
</details>


No site

Links install, modelos, releases.