Pull requests / #1218

#1218 Check the ready-made engine against GitHub's SHA-256 before installing it

closed · @demetree · 0 comments · View on GitHub

Setup & installServer & APIAMD / HIPDocumentationWindows

Description

Check the ready-made engine against GitHub's SHA-256 before installing it

`get_prebuilt()` installed the engine on nothing but the byte count matching the server's
`Content-Length`. A file of the same length but different bytes passed that, and the engine directory then
held code that nothing had checked. This is the install-path half of the hash check the update path already
does (#1162); it is a different code path, so it is a separate change.

**What it does.** Reads the asset's size and SHA-256 from the releases API — a different origin from the
download — and hashes the file setup just downloaded. It happens **before the archive is opened**, so a file
that does not match never reaches `_unpack`, let alone `engine/`. Reuses `verify_sha256()`, the function the
Unsloth shards already use, so the idiom and the finish mark are the ones the project has.

**Measured, not asserted.** On v0.1.40's `strata-windows-x64.zip`: the API reported 131,707,082 bytes and
`cd264b2125fdb85e84a8264da2ab2343463e6c7f9a12f317d4ce7192cd2c6b33`. The download hashed to the same and
installed. A file of exactly 131,707,082 bytes with different content — which the old size check cannot see —
was refused and deleted, leaving the engine directory empty.

**The tag comes out of the URL**, so the exact release the bytes claim to come from is the one that is
checked, not "latest" (#214 tries the checkout's own release first). And the hash is read from
`api.github.com` even when `--prebuilt` points the *download* somewhere else, which is the point: a
compromised mirror cannot supply bytes with a matching digest.

**A wrong file is deleted**, so the next run downloads the published one again, and the verified hash is kept
in the download's `.done` mark so the ~190 MB is not hashed twice.

**If GitHub will not answer, setup stops** rather than install unchecked. The usual cause is the anonymous
rate limit — 60 requests an hour, per internet address, so a shared or office connection runs out — and
`docs/TROUBLESHOOTING.md` now says so. `STRATA_ALLOW_UNVERIFIED_ENGINE=1` is the explicit, recorded way to
accept an unverified engine, for an air-gapped or mirrored install. Nothing is silent either way.

**What this does not cover**, because the limit should be written down rather than implied: it proves the
bytes are the ones GitHub published *for that asset*. That catches a corrupted transfer, a substituted or
mirrored download, a TLS-terminating proxy and a hostile network. It does **not** make a malicious release
safe — if whoever can publish a release publishes a hostile engine, the published hash matches it. Only a
hash pinned in the source closes that, at the cost of a reviewed commit per release. `get_prebuilt()` had no
check at all before this, so this is strictly more than the install path did; pinning remains strictly more
than this does.

`tools/test_setup_engine_hash.py` — 16 tests, no network. Covers the tag/latest/mirror URL shapes, no digest,
a non-SHA-256 digest, a missing asset, GitHub unreachable, the match, the wrong hash (refused **and**
deleted), the wrong size, the hash not being repeated on a second run, the refusal when there is no digest,
and the opt-in escape hatch. Two of them assert the ordering — that the check runs before `ZipFile` and
before `_unpack` — so moving it later fails the suite rather than quietly weakening it.

`tools/test_setup_pins.py`: the five tests that mock the download now supply a digest, which is what a
test mocking the download has to do once the download path verifies it. Their subject is unchanged — which
release and which asset are chosen.

`python -m unittest tools.test_setup_pins` 18 pass. `tools/`: 23 of 24 files pass; `test_setup_choices`
fails on this machine both before and after this change, in `build_engine_hip`, because it wants a C++
compiler (`cl.exe`/`hipcc`) that is not installed — verified against pristine upstream `setup.py`.
`pytest serve` 444 passed, 8 skipped.

Note this is a behaviour change on the install path: a machine that cannot reach the API now stops rather
than installing, and the message says why.

Related on strata.com

Editorial links to help you install, pick models, or read release notes — not part of the upstream thread.