Pull requests / #2544

#2544 calibrate: each step says which value it kept, and why the default stays over a round's best (#2465)

closed · @qiuzan2001 · 0 コメント · GitHub で見る

BenchmarksSetup & installDocumentationLinux

本文

Addresses #2465 (the part that is a bug in what calibrate prints; the choices it makes are unchanged).

**What it changes.** `tools/calibrate.py` keeps a round's best only when the confirm round (the best against the
default, interleaved) or the worker/expert-tier measurements show it more than 3% faster (`pick`, `MIN_GAIN`). That
rule printed nothing, so in #2465 the arrows said PCIe share 1, draft floor 0.7 and the 160-swap tier while the saved
config carried the defaults. Each of the three decisions now ends with one line:

```
    the best against the default, median of 3: the default (PCIe share 0.55 and draft floor 0.5) 125.8, the best (PCIe share 1 and draft floor 0.7) 128.1
    -> PCIe share 1 and draft floor 0.7 measured +1.8% over the default (PCIe share 0.55 and draft floor 0.5), not more than the 3% a setting must win by: the default stays
    ...
    -> every 1, 160 swaps, decay 0.97 measured +2.1% over the default (the engine's expert tier), not more than the 3% a setting must win by: the default stays
```

(numbers here are illustrative, from the issue's medians), or `-> <value>: +N% over the default (...): kept`, or
`-> <default> (the default) is the fastest: kept`. One sentence in docs/DETAILS.md ("Tuning for your PC") says so.

**What it leaves alone.** Which settings are chosen and written: the diff only adds `say()` lines and the `verdict()`
helper, which reads the same medians `pick()` does and returns no line when the default did not start. The 3% rule
itself, the sweep arrows, setup's `calibrate_config()` summary line. Whether the 3% rule is the right one (the issue's
other suggestion, "save exactly what the arrows name") is a maintainer decision, so I did not change it.

**How it was checked.**
- New tests in `tools/test_calibrate.py`: `test_a_round_winner_the_config_does_not_carry_is_explained_2465` (the
  issue's case: PCIe 1 +2%, draft floor 0.7 +0.5%, the 160-swap tier +2%: settings `{}`, and the lines say the
  default stays with the gain; plus a kept worker count and a kept PCIe/draft pair) and `test_verdict` (each branch,
  and a default that measured 0 tok/s is not divided by). Both fail on `main`.
- `python3 tools/test_calibrate.py`: 19 tests OK. `python3 tools/test_setup_choices.py`: OK.
  `tools/test_setup_golden.py` has 8 failures (the unsloth UD-Q4_K_XL rows) on `upstream/main` without this change too,
  the same 8 with it; calibrate is not part of those.
- Python 3.14, Ubuntu (Linux 7.0); no GPU needed (the tests use the stand-in engine). Not run against a real engine.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

関連リンク

インストール・モデル・リリースへの站内リンク。