Pull requests / #1218

#1218 Check the ready-made engine against GitHub's SHA-256 before installing it

closed · @demetree · 0 commentaires · Sur GitHub

Setup & installServer & APIAMD / HIPDocumentationWindows

Description

Check the ready-made engine against GitHub's SHA-256 before installing it

`get_prebuilt()` installed the engine on nothing but the byte count matching the server's
`Content-Length`. A file of the same length but different bytes passed that, and the engine directory then
held code that nothing had checked. This is the install-path half of the hash check the update path already
does (#1162); it is a different code path, so it is a separate change.

**What it does.** Reads the asset's size and SHA-256 from the releases API — a different origin from the
download — and hashes the file setup just downloaded. It happens **before the archive is opened**, so a file
that does not match never reaches `_unpack`, let alone `engine/`. Reuses `verify_sha256()`, the function the
Unsloth shards already use, so the idiom and the finish mark are the ones the project has.

**Measured, not asserted.** On v0.1.40's `strata-windows-x64.zip`: the API reported 131,707,082 bytes and
`cd264b2125fdb85e84a8264da2ab2343463e6c7f9a12f317d4ce7192cd2c6b33`. The download hashed to the same and
installed. A file of exactly 131,707,082 bytes with different content — which the old size check cannot see —
was refused and deleted, leaving the engine directory empty.

**The tag comes out of the URL**, so the exact release the bytes claim to come from is the one that is
checked, not "latest" (#214 tries the checkout's own release first). And the hash is read from
`api.github.com` even when `--prebuilt` points the *download* somewhere else, which is the point: a
compromised mirror cannot supply bytes with a matching digest.

**A wrong file is deleted**, so the next run downloads the published one again, and the verified hash is kept
in the download's `.done` mark so the ~190 MB is not hashed twice.

**If GitHub will not answer, setup stops** rather than install unchecked. The usual cause is the anonymous
rate limit — 60 requests an hour, per internet address, so a shared or office connection runs out — and
`docs/TROUBLESHOOTING.md` now says so. `STRATA_ALLOW_UNVERIFIED_ENGINE=1` is the explicit, recorded way to
accept an unverified engine, for an air-gapped or mirrored install. Nothing is silent either way.

**What this does not cover**, because the limit should be written down rather than implied: it proves the
bytes are the ones GitHub published *for that asset*. That catches a corrupted transfer, a substituted or
mirrored download, a TLS-terminating proxy and a hostile network. It does **not** make a malicious release
safe — if whoever can publish a release publishes a hostile engine, the published hash matches it. Only a
hash pinned in the source closes that, at the cost of a reviewed commit per release. `get_prebuilt()` had no
check at all before this, so this is strictly more than the install path did; pinning remains strictly more
than this does.

`tools/test_setup_engine_hash.py` — 16 tests, no network. Covers the tag/latest/mirror URL shapes, no digest,
a non-SHA-256 digest, a missing asset, GitHub unreachable, the match, the wrong hash (refused **and**
deleted), the wrong size, the hash not being repeated on a second run, the refusal when there is no digest,
and the opt-in escape hatch. Two of them assert the ordering — that the check runs before `ZipFile` and
before `_unpack` — so moving it later fails the suite rather than quietly weakening it.

`tools/test_setup_pins.py`: the five tests that mock the download now supply a digest, which is what a
test mocking the download has to do once the download path verifies it. Their subject is unchanged — which
release and which asset are chosen.

`python -m unittest tools.test_setup_pins` 18 pass. `tools/`: 23 of 24 files pass; `test_setup_choices`
fails on this machine both before and after this change, in `build_engine_hip`, because it wants a C++
compiler (`cl.exe`/`hipcc`) that is not installed — verified against pristine upstream `setup.py`.
`pytest serve` 444 passed, 8 skipped.

Note this is a behaviour change on the install path: a machine that cannot reach the API now stops rather
than installing, and the message says why.

Sur le site

Liens install, modèles, releases.