Pull requests / #3446
#3446 [Benchmark] tools/strata_bench.py: assign the port before the harness starts the server
open · @rodi4eg · 0 コメント · GitHub で見る
BenchmarksSetup & installDocumentationWindowsLinux
本文
## Problem and result
`tools/strata_bench.py` never assigns `port` on the path where the harness starts the server itself (no `--url`). The first use raises `NameError: name 'port' is not defined`, so the run dies before measuring anything:
```
File "tools/strata_bench.py", line 990, in main
say(f"Starting {cfg_path.name} on port {port} (loading takes 1-3 minutes; ...)
NameError: name 'port' is not defined
```
That is the path every community report takes (the harness is started with a config, not against a running server), so the tool cannot measure an install that is not already serving.
With this change the port is assigned once, and the same value is used for the readiness check, the "that port is used by something else" message and `start_server`.
## Related discussion
none
## Approach
One line in `tools/strata_bench.py`:
```python
- base = server_base(cfg.get("host"), a.port or int(cfg.get("port", 8080)))
+ port = a.port or int(cfg.get("port", 8080))
+ base = server_base(cfg.get("host"), port)
```
`port` is then defined for the two later uses in the same branch.
## Validation
**Passed:**
- `python tools/strata_bench.py --config <config> --runs 3` completes the suite (chat, 4K and 16K prompt tests, the agent session) and writes `README.md` + `results.json` + the engine log excerpt. The same command on the parent commit fails with the `NameError` above.
- The `--url http://127.0.0.1:8080` path is unchanged and still works (it never reaches this branch).
- 9 configurations measured back to back with this tool, each a fresh engine per run: 48 arms on one model and 7 on another, with 3 and 5 runs per test.
**Failed:**
- none observed
**Not run:**
- Windows (the fix is in the same Python file; the harness was used on Linux here).
## Limits
none
関連リンク
インストール・モデル・リリースへのサイト内リンク。