Issues / #3561

#3561 [Other] Release artifacts carry no commit, so every standard benchmark from a release is marked nonconforming

open · @atorresg · 0 commentaires · Sur GitHub

BenchmarksSetup & installAMD / HIPModels & quantsLinux

Description

## Summary

A standard benchmark run from a **release** install cannot reach `Status: complete`: the runner cannot record the Strata commit, because no release artifact carries it. The measurements themselves are fine — only the provenance is missing — but `bench_standard.py` treats a missing commit as `nonconforming`, which blocks the community-benchmark submission path for everyone who does not build from source.

The code already expects a release to ship this file. `bench_facts.py:349` says so itself:

> not a git checkout and no COMMIT file in this install; **a release must carry its commit**

So this looks like an omission in the release packaging.

## What the runner does

`strata_cli/bench_facts.py:318`:

```python
def git_commit(folder: Path = ROOT) -> str | None:
    """The full commit of this checkout, or the COMMIT file a release archive carries; None for neither."""
    if (folder / ".git").exists():
        out = run_cmd(["git", "-C", str(folder), "rev-parse", "HEAD"]).strip().lower()
        if HEX40.match(out):
            return out
    try:
        text = (folder / "COMMIT").read_text(encoding="utf-8").strip().lower()
    except OSError:
        return None
    return text if HEX40.match(text) else None
```

`ROOT` is the installed terminal package (`~/.local/share/strata/terminal/<version>`), and it looks for a `.git` directory or a `COMMIT` file containing a 40-hex string. With neither, `software()` records `strata_commit: null` and `runner_commit: null` (`bench_facts.py:340-350`), and `bench_standard.py:792` reports:

> Why this is not a complete standard result:
>   - the Strata commit is unknown (not a git checkout and no COMMIT file)

## Verified against the v0.2.1 release

No release artifact carries the commit:

| Artifact | Commit present? |
| --- | --- |
| `strata-control-0.1.46.3-macos-arm64.tar.gz` | **No** — no `COMMIT` entry in the archive (it does ship `VERSION`, containing `0.1.46.3`) |
| Engine `BUILD.json` | **No** — only `{"version": "0.1.46"}` |
| Signed feed `strata-feed.json` | **No** — root keys are `app`, `channel`, `control`, `engine`, `expires_at`, `feed`, `generated_at`, `schema_version`, `sequence`; no commit field anywhere |

So there is no in-band source for the runner to fall back to.

## Steps to reproduce

1. Install the Terminal from the release on macOS arm64 (or Linux x86-64): `curl -fsSL https://github.com/Niko1221/Strata/releases/latest/download/install.sh | sh`
2. Set up and save a served profile, then run the standard suite:
   ```sh
   strata profile create myprofile --from-config ~/.local/share/strata/install/strata-q2_0.json
   strata bench run --profile myprofile --track default --track-notes "Strata 0.2.1 release"
   ```
3. The run finishes with every request and sanity check passing, and reports:
   ```
   Status: nonconforming. 21 of 21 required requests finished, 0 failed, 0 skipped.
   ...
   Why this is not a complete standard result:
     - the Strata commit is unknown (not a git checkout and no COMMIT file)
   ```

Observed on macOS 27.0.1, Apple M4 Max, Strata 0.2.1 / terminal 0.1.46.3 / engine 0.1.46-metal. Nothing machine-specific is involved: any release install on any platform gets the same result.

## Workaround

Write the commit into the install directory. For v0.2.1 that is `90767252f420e25223c739d604145ad15911d48b`:

```sh
printf '90767252f420e25223c739d604145ad15911d48b\n' \
  > ~/.local/share/strata/terminal/0.1.46.3/COMMIT
```

The run then reports `Status: complete` with no other configuration change.

Note for whoever writes the fix: the `v0.2.1` tag is an **annotated** tag (`refs/tags/v0.2.1` → tag object `49ac15cbb32a8662602aaa7b8fc61bf6acef4cf0`), so the commit has to be obtained by dereferencing it, not by reading the tag ref directly.

## Impact

Anyone benchmarking from a release cannot produce a `complete` standard result, which is what `COMMUNITY_BENCHMARKS.md` asks contributors to submit. It also forces people to hand-edit their own install, and it makes two otherwise identical runs differ in status for a reason unrelated to the hardware.

#3481 shows the same effect from another machine: one run "had no COMMIT file in the image (nonconforming)", and the accepted report carries a hand-placed commit that "the runner records". So this is already costing contributors a manual step.

Sur le site

Liens install, modèles, releases.