Pull requests / #2601

#2601 docs/COMMUNITY_BENCHMARKS.md: keep the expert profile fixed while measuring, and the measured prompts out of a profile's traces (#2088)

closed · @qiuzan2001 · 0 Kommentare · Auf GitHub

BenchmarksServer & APIDocumentation

Beschreibung

Docs only: one paragraph in `docs/COMMUNITY_BENCHMARKS.md` ("How to measure"), the line #2088 left open.

**What it says:** keep the expert profile fixed while measuring (`--expert-profile FILE`, no profile save); a save, by `--expert-profile-save` or by the config's `"expert_profile_save"`, writes at the server's stop and every 10 minutes; since 0.1.43.1 the bench removes `--expert-profile-save` from a server it starts, but not the config key, and a server you start yourself keeps both; the start-up line names the profile; and a profile built from routing traces must not contain the prompts it is measured with (the second point of #2088).

**What it leaves alone:** no code. The bench still does not remove the config's `"expert_profile_save"` key from a server it starts (`tools/strata_bench.py:668-681` strips only the argument; `serve/server.py:2645-2652` adds the argument back from the key). The paragraph says so instead of changing the tool.

**How it was checked:** each claim read against the code at `296b59d2`: `src/program/generate.cpp:869-872` (save at QUIT and every `--expert-profile-save-every` minutes, default 10) and `:3697` (the start-up line), `serve/server.py:2645-2652`, `tools/strata_bench.py:668-695`, `git tag --contains 017a672a` (v0.1.43.1 and later), and the three tools exist. A second reader checked the claims against the same lines.

Closes #2088 once the maintainer agrees the line is enough.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Mehr auf der Site

Links zu Install, Modellen, Releases.