Issues / #2088
#2088 `strata_bench.py` + `--expert-profile-save`: the bench run overwrites your calibrated live profile with what it learned
open · @mmbeanz · 3 comments · View on GitHub
BenchmarksSetup & installDocumentation
Description
Setup with `--expert-profile X --expert-profile-save X` (the profile-save loop setup offers) on a running-server config. `--expert-profile-save` writes the profile when the server **quits**. `strata_bench.py --url http://127.0.0.1:8080` attaches to the running server but still **shuts it down** at the end of the suite. Result: the benchmark silently replaces the live profile with the one learned from three short chats and word-list prompts — the calibration is gone and nothing says so.
We hit this on a 5060 Ti + 3060 split (v0.1.43, `7695859a`): the first on-box suite run mutated the live profile; the run and its results had to be discarded.
Suggested fixes, cheapest first:
1. Document it in `docs/COMMUNITY_BENCHMARKS.md` ("pin the profile: use `--expert-profile` without `--save` for benching") — one line.
2. Have `strata_bench.py` warn when the attached server's config carries `--expert-profile-save`.
3. Or don't shut down a server it didn't start (`--url` mode), and let the owner manage the lifecycle.
Workaround we used: bench configs point at a plain pinned `data/expert-profile.bin` with no save flag, and the engine log line `profile ...: 24576 ranked pairs, built for 24576 slots` at every boot confirms which profile each run started from.
*Drafted with AI assistance; I reviewed every line and ran the measurements myself.*Related on strata.com
Editorial links to help you install, pick models, or read release notes — not part of the upstream thread.