Pull requests / #2885
#2885 tools: a --compare path that is not a bench report is a message, not a traceback
open · @ctr-137 · 0 Kommentare · Auf GitHub
BenchmarksAMD / HIPNVIDIA / CUDADocumentationWindows
Beschreibung
`--compare` and `--compare-only` are documented in `docs/COMMUNITY_BENCHMARKS.md` and named in the `BENCH.bat` /
`bench.sh` headers, and the two flags are easy to swap. Both went through `load_results()`, which read the path
and let whatever came out of the read escape. Three ways it went wrong, every one of them ending the run as a
traceback:
**A path that is not there** - a typo:
```
$ python tools/strata_bench.py --compare bench/results/typo --compare-only bench/results/also-typo
File "tools/strata_bench.py", line 896, in load_results
return json.loads(p.read_text(encoding="utf-8"))
FileNotFoundError: [Errno 2] No such file or directory: 'bench/results/typo'
```
**A folder that is not a results folder, or a file that is not JSON** - `IsADirectoryError` / `JSONDecodeError`,
the same way.
**A `results.json` that is not a bench report.** This one needs no typo at all:
```
$ python tools/strata_bench.py --compare bench/results/2026-10-01-community-2x-titan-rtx \
--compare-only bench/results/2026-10-04-community-2x-rtx3060
File "tools/strata_bench.py", line 878, in compare_table
a, b = summary_metrics(old["results"]), summary_metrics(new["results"])
TypeError: list indices must be integers or slices, not str
```
`bench/results/` holds `results.json` files from the other tools in `tools/` as well - `ab_engine.py` and the
`conversation_cache_*` ones write a top-level list or a dict of `summary`/`runs`. Of the 27 `results.json` in
`bench/results/` today, none is a `strata_bench.py` report, so `--compare` between two of the folders the repo
itself ships reaches `compare_table()` and dies there.
### Where I changed it
`load_results()` raises `ValueError` carrying the path and what is wrong with it, and the two callers print that
instead of letting it out:
```
ValueError: bench/results/typo has no results.json in it
ValueError: bench/results/typo cannot be read (No such file or directory)
ValueError: bench/results/x/results.json is not valid JSON (...)
ValueError: bench/results/2026-10-01-community-2x-titan-rtx/results.json is not a report tools/strata_bench.py wrote
```
- `--compare-only` reads both folders before it starts anything, so it reports and exits 2, the way the bench
reports its other bad input.
- `--compare` is read *after* the report is written, so it notes that only the comparison was lost rather than
undoing a finished run.
The report check is structural - a dict with a dict `machine` and a dict `results` holding `chat`/`prompts`/
`agent`/`clients`, which is what the tables read - rather than a schema number, so it does not turn a future
schema into a new failure. `test_compare` loads a report the writer really produced, so the check cannot quietly
become too strict without a test going red.
This is the second half of the note at the end of #2792. The first half of that note - a started server left
behind - no longer applies on `main`: `main()` sets `started` immediately after `start_server()` and its `finally`
stops the server on any exception. I checked that rather than assumed it, so I did not touch it.
### What I left alone
`project_version()` still catches only `OSError`, so a `CMakeLists.txt` that cannot be read for any other reason
still reaches the top of `main()` as a traceback. Its neighbour `find_configs()` catches `(OSError, ValueError)`,
so that guard looks like an oversight - but #2879 already fixes the one case that actually bit anyone, and widening
it is a separate change.
### How I checked
Six tests in `tools/test_strata_bench.py`, no GPU or server needed: each way a path is wrong (including a
`results.json` shaped like the other tools'), that a real results folder still loads, both call sites, and that the
comparison of a finished run is only noted rather than fatal.
Five of the six fail on `main` today - four with the raw `FileNotFoundError`, one with `ValueError not raised` for
the other tools' reports. With this, the 23 tests in the file pass (`python -m unittest tools.test_strata_bench`).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Mehr auf der Site
Links zu Install, Modellen, Releases.