Pull requests / #2885

#2885 tools: a --compare path that is not a bench report is a message, not a traceback

open · @ctr-137 · 0 comentários · No GitHub

BenchmarksAMD / HIPNVIDIA / CUDADocumentationWindows

Descrição

`--compare` and `--compare-only` are documented in `docs/COMMUNITY_BENCHMARKS.md` and named in the `BENCH.bat` /
`bench.sh` headers, and the two flags are easy to swap.  Both went through `load_results()`, which read the path
and let whatever came out of the read escape.  Three ways it went wrong, every one of them ending the run as a
traceback:

**A path that is not there** - a typo:

```
$ python tools/strata_bench.py --compare bench/results/typo --compare-only bench/results/also-typo
  File "tools/strata_bench.py", line 896, in load_results
    return json.loads(p.read_text(encoding="utf-8"))
FileNotFoundError: [Errno 2] No such file or directory: 'bench/results/typo'
```

**A folder that is not a results folder, or a file that is not JSON** - `IsADirectoryError` / `JSONDecodeError`,
the same way.

**A `results.json` that is not a bench report.**  This one needs no typo at all:

```
$ python tools/strata_bench.py --compare bench/results/2026-10-01-community-2x-titan-rtx \
                               --compare-only bench/results/2026-10-04-community-2x-rtx3060
  File "tools/strata_bench.py", line 878, in compare_table
    a, b = summary_metrics(old["results"]), summary_metrics(new["results"])
TypeError: list indices must be integers or slices, not str
```

`bench/results/` holds `results.json` files from the other tools in `tools/` as well - `ab_engine.py` and the
`conversation_cache_*` ones write a top-level list or a dict of `summary`/`runs`.  Of the 27 `results.json` in
`bench/results/` today, none is a `strata_bench.py` report, so `--compare` between two of the folders the repo
itself ships reaches `compare_table()` and dies there.

### Where I changed it

`load_results()` raises `ValueError` carrying the path and what is wrong with it, and the two callers print that
instead of letting it out:

```
ValueError: bench/results/typo has no results.json in it
ValueError: bench/results/typo cannot be read (No such file or directory)
ValueError: bench/results/x/results.json is not valid JSON (...)
ValueError: bench/results/2026-10-01-community-2x-titan-rtx/results.json is not a report tools/strata_bench.py wrote
```

- `--compare-only` reads both folders before it starts anything, so it reports and exits 2, the way the bench
  reports its other bad input.
- `--compare` is read *after* the report is written, so it notes that only the comparison was lost rather than
  undoing a finished run.

The report check is structural - a dict with a dict `machine` and a dict `results` holding `chat`/`prompts`/
`agent`/`clients`, which is what the tables read - rather than a schema number, so it does not turn a future
schema into a new failure.  `test_compare` loads a report the writer really produced, so the check cannot quietly
become too strict without a test going red.

This is the second half of the note at the end of #2792.  The first half of that note - a started server left
behind - no longer applies on `main`: `main()` sets `started` immediately after `start_server()` and its `finally`
stops the server on any exception.  I checked that rather than assumed it, so I did not touch it.

### What I left alone

`project_version()` still catches only `OSError`, so a `CMakeLists.txt` that cannot be read for any other reason
still reaches the top of `main()` as a traceback.  Its neighbour `find_configs()` catches `(OSError, ValueError)`,
so that guard looks like an oversight - but #2879 already fixes the one case that actually bit anyone, and widening
it is a separate change.

### How I checked

Six tests in `tools/test_strata_bench.py`, no GPU or server needed: each way a path is wrong (including a
`results.json` shaped like the other tools'), that a real results folder still loads, both call sites, and that the
comparison of a finished run is only noted rather than fatal.

Five of the six fail on `main` today - four with the raw `FileNotFoundError`, one with `ValueError not raised` for
the other tools' reports.  With this, the 23 tests in the file pass (`python -m unittest tools.test_strata_bench`).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

No site

Links install, modelos, releases.