Pull requests / #1218
#1218 Check the ready-made engine against GitHub's SHA-256 before installing it
closed · @demetree · 0 comments · View on GitHub
Setup & installServer & APIAMD / HIPDocumentationWindows
Description
Check the ready-made engine against GitHub's SHA-256 before installing it `get_prebuilt()` installed the engine on nothing but the byte count matching the server's `Content-Length`. A file of the same length but different bytes passed that, and the engine directory then held code that nothing had checked. This is the install-path half of the hash check the update path already does (#1162); it is a different code path, so it is a separate change. **What it does.** Reads the asset's size and SHA-256 from the releases API — a different origin from the download — and hashes the file setup just downloaded. It happens **before the archive is opened**, so a file that does not match never reaches `_unpack`, let alone `engine/`. Reuses `verify_sha256()`, the function the Unsloth shards already use, so the idiom and the finish mark are the ones the project has. **Measured, not asserted.** On v0.1.40's `strata-windows-x64.zip`: the API reported 131,707,082 bytes and `cd264b2125fdb85e84a8264da2ab2343463e6c7f9a12f317d4ce7192cd2c6b33`. The download hashed to the same and installed. A file of exactly 131,707,082 bytes with different content — which the old size check cannot see — was refused and deleted, leaving the engine directory empty. **The tag comes out of the URL**, so the exact release the bytes claim to come from is the one that is checked, not "latest" (#214 tries the checkout's own release first). And the hash is read from `api.github.com` even when `--prebuilt` points the *download* somewhere else, which is the point: a compromised mirror cannot supply bytes with a matching digest. **A wrong file is deleted**, so the next run downloads the published one again, and the verified hash is kept in the download's `.done` mark so the ~190 MB is not hashed twice. **If GitHub will not answer, setup stops** rather than install unchecked. The usual cause is the anonymous rate limit — 60 requests an hour, per internet address, so a shared or office connection runs out — and `docs/TROUBLESHOOTING.md` now says so. `STRATA_ALLOW_UNVERIFIED_ENGINE=1` is the explicit, recorded way to accept an unverified engine, for an air-gapped or mirrored install. Nothing is silent either way. **What this does not cover**, because the limit should be written down rather than implied: it proves the bytes are the ones GitHub published *for that asset*. That catches a corrupted transfer, a substituted or mirrored download, a TLS-terminating proxy and a hostile network. It does **not** make a malicious release safe — if whoever can publish a release publishes a hostile engine, the published hash matches it. Only a hash pinned in the source closes that, at the cost of a reviewed commit per release. `get_prebuilt()` had no check at all before this, so this is strictly more than the install path did; pinning remains strictly more than this does. `tools/test_setup_engine_hash.py` — 16 tests, no network. Covers the tag/latest/mirror URL shapes, no digest, a non-SHA-256 digest, a missing asset, GitHub unreachable, the match, the wrong hash (refused **and** deleted), the wrong size, the hash not being repeated on a second run, the refusal when there is no digest, and the opt-in escape hatch. Two of them assert the ordering — that the check runs before `ZipFile` and before `_unpack` — so moving it later fails the suite rather than quietly weakening it. `tools/test_setup_pins.py`: the five tests that mock the download now supply a digest, which is what a test mocking the download has to do once the download path verifies it. Their subject is unchanged — which release and which asset are chosen. `python -m unittest tools.test_setup_pins` 18 pass. `tools/`: 23 of 24 files pass; `test_setup_choices` fails on this machine both before and after this change, in `build_engine_hip`, because it wants a C++ compiler (`cl.exe`/`hipcc`) that is not installed — verified against pristine upstream `setup.py`. `pytest serve` 444 passed, 8 skipped. Note this is a behaviour change on the install path: a machine that cannot reach the API now stops rather than installing, and the message says why.
Related on strata.com
Editorial links to help you install, pick models, or read release notes — not part of the upstream thread.