Pull requests / #1162
#1162 Verify the downloaded engine against GitHub's SHA-256 (replaces #645)
closed · @demetree · 0 comentarios · En GitHub
Setup & installServer & APINVIDIA / CUDAWindows
Descripción
## Verify the download against GitHub's SHA-256 (and rebase onto current `main`) Closes the review objection: without a hash check, the only thing between a substituted download and a running engine was the byte count, which a file of the same length passes. `setup.py`'s `get_prebuilt()` has the same gap today - it calls `download()` and nothing else. **GitHub publishes the hash**, so this needed no release-process change. The releases API returns `digest` (`sha256:<hex>`) for every asset on every release - I checked v0.1.34 through v0.1.40.1 and every one is present, including the ones published before GitHub added the field. This is the same idea as `setup.py`'s `verify_sha256()`, which pins hashes for the Unsloth shards in the repo; the API's `digest` is the equivalent for a published asset. It is worth having because the two values arrive on different connections: the bytes from the release download, the expected hash from `api.github.com`. A substituted download does not come with a matching hash. What it does now: - computes the hash **in the same pass as the download**, so a 190 MB asset is read once; - **deletes** a file that does not match, and names both hashes in the error; - **refuses** a release that publishes no digest, rather than installing unchecked - and says `UPDATE.bat` will still fetch it, so nobody is stuck; - does not accept `sha512:` or `md5:` as if it were sha256; - reports "SHA-256 verified" on the step, and puts the hash that arrived in the state. ### Measured, against the live release On this machine (GTX 1070, compute capability 6.1, CUDA 12.6 engine), a real update of the real v0.1.40.1: ``` Download the engine 190.2 MB, SHA-256 verified Inspect the archive 3 files, engine 0.1.40 for v0.1.40.1 Run the staged engine it starts and prints its usage Start it and check the version running 0.1.40 ``` 18 s end to end. The digest came from the API as `5ffaf2ba1fc68cc4cc7ef26889a0eaad5b302e251067a12f191911ea29770f42` and matched. ### Also fixed: the asset table had gone stale It was a hardcoded map, and it did not know about `strata-windows-x64-cuda12.zip` (`setup.py`'s `CUDA12_ASSET`, published since). A machine on the experimental CUDA 12 engine would have been handed the CUDA 13 build, which it cannot run - and on this machine that is exactly what would have happened. The name is now built from the platform plus **the CUDA major the installed engine's own `BUILD.json` records**, so keeping an install on the CUDA line it already uses needs no code change when a new asset appears, and an unknown CUDA falls back rather than refusing. ### Two bugs the live run found that no offline test had **The tag is not the engine version.** `v0.1.40.1` is a hotfix release whose engine is `v0.1.40`, so the exact comparison in the inspect step refused a perfectly good update. Only an *older* engine is a refusal now - that means the wrong asset was picked. `_verify()` had the same comparison and the same bug: it failed *after* a correct install and rolled back, which is the right behaviour for the wrong reason. The panel repeated it too, saying "The engine is now v0.1.40.1" about a 0.1.40 engine. ### Please note: this is a new branch `main` has moved ~1150 commits and its history was rewritten - the previous branch (#645) shared **no ancestry** with upstream at all, so its diff was not reviewable. This is a fresh branch off `82f46a8` with the work re-applied, not a rebase. #645 should be closed in favour of this. ### The UI port nearly broke the app, and nothing noticed Splicing the panel into `app.js` with PowerShell line arrays silently dropped ~30 lines, including `fmt`, `kfmt` and `gb`. `node --check` passed - the file still parsed. Same lesson as `vv0.1.38`: a syntax check is not a functional check. The insert is now done in Python and verified by asserting that every top-level name upstream defines still exists afterwards, and the app was re-checked in a browser. The same re-encode turned the step marks into mojibake (`ľó`) in the browser; the block is now fetched as raw bytes and the render check asserts the code points. ### Tests ``` python serve/test_update.py 110 checks python serve/test_update_http.py 51 checks, real HTTP python -m pytest serve 444 passed, 8 skipped ``` The hash cases are asserted as firmly as the rest: a matching digest updates and reports the hash; a same-length file with different bytes is refused, deleted, and **no later step runs**; no digest is refused; a non-sha256 digest is refused; a four-part tag with a three-part engine is accepted while an older engine is refused; and the same refusal over HTTP. The UI render check (61 checks) is **not** in the PR - it needs node and the repo has no JS harness. ### Still open from before, unchanged `BUILD.json` with a BOM (`json.loads(read_text())` at `serve/server.py` raises on it, so an engine saved by a Windows editor reports no version). One line, `encoding="utf-8-sig"`. I would rather send it as its own PR than smuggle it in here. `get_prebuilt()` could use the same `digest` check described above - that is a change in `setup.py` and I have not touched it.
En el sitio
Enlaces a install, modelos, releases.