#6140 [Benchmark] ab_engine: compare Strata versions with their own defaults, isolated from the install

closed · @xidus90 · 0 评论 · 去 GitHub 看

BenchmarksSetup & installAMD / HIPNVIDIA / CUDAModels & quantsSecurityWindowsLinux

说明

## Problem

`terminal/tools/ab_engine.py` compares two engine builds on one config. It cannot answer the question users ask after an update: "is the new release faster than the old one on my PC, with the settings each release picks?" Doing that by hand means swapping installs, and it is easy to change the real install or its settings on the way.

## Result

A new, opt-in mode compares Strata releases. Each version runs as a user would have got it: the engine and the control of the same release, with that control's own defaults, optionally tuned once. The normal install is not used and not changed, and the tool checks that before and after.

```
python terminal/tools/ab_engine.py --model-path <gguf|folder> --engines 0.1.57..0.1.60 [--rounds 5] [--tune quick|thorough]
```

The run writes `results.json`, `report.html` (a sortable table and a card) and `report.png` (the card, from the same HTML through headless Edge/Chrome/Chromium) to `~/strata-ab/runs/<timestamp>/`. The configs each release generated, tuned ones included, are kept in `~/strata-ab/configs/<version>/<timestamp>/`. The tool deletes nothing in its folder.

The old call `ab_engine.py <config> <arms…>` is unchanged.

## Approach

- `terminal/tools/ab_versions.py`:
  - reads the GitHub releases (no token) and pairs `strata-engine-<ver>` with `strata-control-<ver>` of the same release;
  - downloads both and checks them against that release's `SHA256SUMS.txt`, then unpacks them with `updater.extract` into a cache. A cut download leaves nothing that looks like a cache hit;
  - builds a fresh home per version and run. `HOME`, `USERPROFILE`, `APPDATA`, `LOCALAPPDATA` and the XDG folders point into it, `STRATA_*` is removed from the environment, and the engine and control are laid out the way the updater does it.
- The release's own control starts the server: `strata launch --model-path … --yes --no-tui --output json`. On Windows it runs through the control's bundled Python, because `strata.cmd` mangles paths with `&`, `%` or `^`. Every launch attempt is followed by `launch --stop`, also on failure and on Ctrl-C.
- After each start the tool reads the `engine started:` line. If a control served a different engine than the one asked for, that version is reported as not run, with the reason, and gives no numbers.
- Guards:
  - the tool refuses to start beside a running Strata server or on a port in use;
  - it takes a size and modification-time list of the real Strata folders (and of a Strata data folder holding the model) before and after the run, and exits 1 if anything changed;
  - a crash in one version skips only that version.
- Measurement is the same as the old mode: same prompts, rounds alternating the order, medians.
- `terminal/tools/ab_report.py` builds the HTML and the PNG card.
- It uses only the standard library and modules that are already in the repo.

## Validation

**Passed**
- `python -m pytest terminal/tools/test_ab_versions.py terminal/tools/test_ab_report.py terminal/tools/test_ab_engine.py`: 75 passed on Python 3.13.17. The same 75 pass on 3.11.16 (unittest). No GPU or network is used.
- Old mode against a fake server, before and after this change: `ab_engine.jsonl` and the printed output are byte-identical. Only the argparse usage text differs, because of the new options.
- Real run 1, 2026-11-16, RTX 3090 24 GB, Ryzen 7 9800X3D, 128 GiB RAM, Windows 11:
  - model: Qwen3.8-Flash-Next SC117 abliterated UD-IQ4_XS-v31;
  - setup: 0.1.59 against 0.1.60, `--tune quick`, 2 rounds;
  - both versions ran and each tuned once;
  - first launch: 781 s and 1039 s; restarts: 62 s and 59 s.
- Real run 2, 2026-11-17, same PC and model, 0.1.57 to 0.1.60, 5 rounds, not tuned. Medians of rounds 2 to 5; the cold first round is left out:

| engine | released | decode tok/s (mean of 4 requests) | long prompt tok/s (6,345 tokens) | restart s |
|---|---|---|---|---|
| 0.1.57 | v0.2.11 | 76.4 | 2139 | 58.0 |
| 0.1.58 | v0.2.12 | 77.1 | 2134 | 58.2 |
| 0.1.59 | v0.2.13 | 77.8 | 2128 | 58.5 |
| 0.1.60 | v0.2.14 | 75.8 | 2508 | 56.2 |

  On this PC the long-prompt read is 17% faster in 0.1.60 (all four counted rounds: 2353 to 2521 tok/s, against 2092 to 2158 for the three older versions). Decode does not differ beyond the round-to-round spread of about ±5%.

**Failed, or found by the tool**
- 0.1.48 to 0.1.56 cannot use the UD-IQ4_XS-v31 file. Their pack tool says `… is Q8_0, but the engine requires BF16`, and support for this model arrived in v0.2.11. The tool reported each one as not run, with the reason, and went on.
- The untouched check flagged files twice:
  - run 1: three `…-0000N-of-00004.gguf.done` files next to the model shards, written by a release control during the run;
  - both runs: `~/.strata/oauth-hb/refresh.log`, which belongs to another program that also writes under `~/.strata`.
- `report.png` was missing once, because Edge exited before writing the screenshot. Fixed in the last commit by waiting for the file and trying twice; checked by hand on the same run.

**Not run**
- Linux, macOS, AMD/HIP, Intel/SYCL, more than one GPU.
- `--tune thorough`.
- A run with a standard catalog model (for example UD-Q4_K_XL) across older releases.

## Limits

- The card's medians include round 0 (the cold first start), so with 2 rounds the card can show a gain that is really the cold start. The table in this PR leaves round 0 out; the card does not yet.
- `~/.strata` is watched as a whole, so other programs that write there give a false alarm.
- Each run copies every engine into its own home, so disk use grows. Old homes under `~/strata-ab/homes` are deleted by hand.
- On NVIDIA without `--backend`, the CUDA 13 build is used when a release has both builds. That is the same tie-break as `feed.select_engine`. An older driver needs `--backend cuda-cuda12`.

No issue was opened before this PR. I'm happy to move the discussion to one if you prefer that for a feature of this size.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

本站相关内容

相关页面的快捷入口。