Pull requests / #1218
#1218 Check the ready-made engine against GitHub's SHA-256 before installing it
closed · @demetree · 0 Kommentare · Auf GitHub
Setup & installServer & APIAMD / HIPDocumentationWindows
Beschreibung
Check the ready-made engine against GitHub's SHA-256 before installing it `get_prebuilt()` installed the engine on nothing but the byte count matching the server's `Content-Length`. A file of the same length but different bytes passed that, and the engine directory then held code that nothing had checked. This is the install-path half of the hash check the update path already does (#1162); it is a different code path, so it is a separate change. **What it does.** Reads the asset's size and SHA-256 from the releases API — a different origin from the download — and hashes the file setup just downloaded. It happens **before the archive is opened**, so a file that does not match never reaches `_unpack`, let alone `engine/`. Reuses `verify_sha256()`, the function the Unsloth shards already use, so the idiom and the finish mark are the ones the project has. **Measured, not asserted.** On v0.1.40's `strata-windows-x64.zip`: the API reported 131,707,082 bytes and `cd264b2125fdb85e84a8264da2ab2343463e6c7f9a12f317d4ce7192cd2c6b33`. The download hashed to the same and installed. A file of exactly 131,707,082 bytes with different content — which the old size check cannot see — was refused and deleted, leaving the engine directory empty. **The tag comes out of the URL**, so the exact release the bytes claim to come from is the one that is checked, not "latest" (#214 tries the checkout's own release first). And the hash is read from `api.github.com` even when `--prebuilt` points the *download* somewhere else, which is the point: a compromised mirror cannot supply bytes with a matching digest. **A wrong file is deleted**, so the next run downloads the published one again, and the verified hash is kept in the download's `.done` mark so the ~190 MB is not hashed twice. **If GitHub will not answer, setup stops** rather than install unchecked. The usual cause is the anonymous rate limit — 60 requests an hour, per internet address, so a shared or office connection runs out — and `docs/TROUBLESHOOTING.md` now says so. `STRATA_ALLOW_UNVERIFIED_ENGINE=1` is the explicit, recorded way to accept an unverified engine, for an air-gapped or mirrored install. Nothing is silent either way. **What this does not cover**, because the limit should be written down rather than implied: it proves the bytes are the ones GitHub published *for that asset*. That catches a corrupted transfer, a substituted or mirrored download, a TLS-terminating proxy and a hostile network. It does **not** make a malicious release safe — if whoever can publish a release publishes a hostile engine, the published hash matches it. Only a hash pinned in the source closes that, at the cost of a reviewed commit per release. `get_prebuilt()` had no check at all before this, so this is strictly more than the install path did; pinning remains strictly more than this does. `tools/test_setup_engine_hash.py` — 16 tests, no network. Covers the tag/latest/mirror URL shapes, no digest, a non-SHA-256 digest, a missing asset, GitHub unreachable, the match, the wrong hash (refused **and** deleted), the wrong size, the hash not being repeated on a second run, the refusal when there is no digest, and the opt-in escape hatch. Two of them assert the ordering — that the check runs before `ZipFile` and before `_unpack` — so moving it later fails the suite rather than quietly weakening it. `tools/test_setup_pins.py`: the five tests that mock the download now supply a digest, which is what a test mocking the download has to do once the download path verifies it. Their subject is unchanged — which release and which asset are chosen. `python -m unittest tools.test_setup_pins` 18 pass. `tools/`: 23 of 24 files pass; `test_setup_choices` fails on this machine both before and after this change, in `build_engine_hip`, because it wants a C++ compiler (`cl.exe`/`hipcc`) that is not installed — verified against pristine upstream `setup.py`. `pytest serve` 444 passed, 8 skipped. Note this is a behaviour change on the install path: a machine that cannot reach the API now stops rather than installing, and the message says why.
Mehr auf der Site
Links zu Install, Modellen, Releases.