Issues / #3561
#3561 [Other] Release artifacts carry no commit, so every standard benchmark from a release is marked nonconforming
open · @atorresg · 0 comentários · No GitHub
BenchmarksSetup & installAMD / HIPModels & quantsLinux
Descrição
## Summary
A standard benchmark run from a **release** install cannot reach `Status: complete`: the runner cannot record the Strata commit, because no release artifact carries it. The measurements themselves are fine — only the provenance is missing — but `bench_standard.py` treats a missing commit as `nonconforming`, which blocks the community-benchmark submission path for everyone who does not build from source.
The code already expects a release to ship this file. `bench_facts.py:349` says so itself:
> not a git checkout and no COMMIT file in this install; **a release must carry its commit**
So this looks like an omission in the release packaging.
## What the runner does
`strata_cli/bench_facts.py:318`:
```python
def git_commit(folder: Path = ROOT) -> str | None:
"""The full commit of this checkout, or the COMMIT file a release archive carries; None for neither."""
if (folder / ".git").exists():
out = run_cmd(["git", "-C", str(folder), "rev-parse", "HEAD"]).strip().lower()
if HEX40.match(out):
return out
try:
text = (folder / "COMMIT").read_text(encoding="utf-8").strip().lower()
except OSError:
return None
return text if HEX40.match(text) else None
```
`ROOT` is the installed terminal package (`~/.local/share/strata/terminal/<version>`), and it looks for a `.git` directory or a `COMMIT` file containing a 40-hex string. With neither, `software()` records `strata_commit: null` and `runner_commit: null` (`bench_facts.py:340-350`), and `bench_standard.py:792` reports:
> Why this is not a complete standard result:
> - the Strata commit is unknown (not a git checkout and no COMMIT file)
## Verified against the v0.2.1 release
No release artifact carries the commit:
| Artifact | Commit present? |
| --- | --- |
| `strata-control-0.1.46.3-macos-arm64.tar.gz` | **No** — no `COMMIT` entry in the archive (it does ship `VERSION`, containing `0.1.46.3`) |
| Engine `BUILD.json` | **No** — only `{"version": "0.1.46"}` |
| Signed feed `strata-feed.json` | **No** — root keys are `app`, `channel`, `control`, `engine`, `expires_at`, `feed`, `generated_at`, `schema_version`, `sequence`; no commit field anywhere |
So there is no in-band source for the runner to fall back to.
## Steps to reproduce
1. Install the Terminal from the release on macOS arm64 (or Linux x86-64): `curl -fsSL https://github.com/Niko1221/Strata/releases/latest/download/install.sh | sh`
2. Set up and save a served profile, then run the standard suite:
```sh
strata profile create myprofile --from-config ~/.local/share/strata/install/strata-q2_0.json
strata bench run --profile myprofile --track default --track-notes "Strata 0.2.1 release"
```
3. The run finishes with every request and sanity check passing, and reports:
```
Status: nonconforming. 21 of 21 required requests finished, 0 failed, 0 skipped.
...
Why this is not a complete standard result:
- the Strata commit is unknown (not a git checkout and no COMMIT file)
```
Observed on macOS 27.0.1, Apple M4 Max, Strata 0.2.1 / terminal 0.1.46.3 / engine 0.1.46-metal. Nothing machine-specific is involved: any release install on any platform gets the same result.
## Workaround
Write the commit into the install directory. For v0.2.1 that is `90767252f420e25223c739d604145ad15911d48b`:
```sh
printf '90767252f420e25223c739d604145ad15911d48b\n' \
> ~/.local/share/strata/terminal/0.1.46.3/COMMIT
```
The run then reports `Status: complete` with no other configuration change.
Note for whoever writes the fix: the `v0.2.1` tag is an **annotated** tag (`refs/tags/v0.2.1` → tag object `49ac15cbb32a8662602aaa7b8fc61bf6acef4cf0`), so the commit has to be obtained by dereferencing it, not by reading the tag ref directly.
## Impact
Anyone benchmarking from a release cannot produce a `complete` standard result, which is what `COMMUNITY_BENCHMARKS.md` asks contributors to submit. It also forces people to hand-edit their own install, and it makes two otherwise identical runs differ in status for a reason unrelated to the hardware.
#3481 shows the same effect from another machine: one run "had no COMMIT file in the image (nonconforming)", and the accepted report carries a hand-placed commit that "the runner records". So this is already costing contributors a manual step.
No site
Links install, modelos, releases.