Pull requests / #1162

#1162 Verify the downloaded engine against GitHub's SHA-256 (replaces #645)

closed · @demetree · 0 comentarios · En GitHub

Setup & installServer & APINVIDIA / CUDAWindows

Descripción

## Verify the download against GitHub's SHA-256 (and rebase onto current `main`)

Closes the review objection: without a hash check, the only thing between a substituted download and a
running engine was the byte count, which a file of the same length passes. `setup.py`'s `get_prebuilt()`
has the same gap today - it calls `download()` and nothing else.

**GitHub publishes the hash**, so this needed no release-process change. The releases API returns
`digest` (`sha256:<hex>`) for every asset on every release - I checked v0.1.34 through v0.1.40.1 and every
one is present, including the ones published before GitHub added the field. This is the same idea as
`setup.py`'s `verify_sha256()`, which pins hashes for the Unsloth shards in the repo; the API's `digest` is
the equivalent for a published asset.

It is worth having because the two values arrive on different connections: the bytes from the release
download, the expected hash from `api.github.com`. A substituted download does not come with a matching
hash.

What it does now:

- computes the hash **in the same pass as the download**, so a 190 MB asset is read once;
- **deletes** a file that does not match, and names both hashes in the error;
- **refuses** a release that publishes no digest, rather than installing unchecked - and says `UPDATE.bat`
  will still fetch it, so nobody is stuck;
- does not accept `sha512:` or `md5:` as if it were sha256;
- reports "SHA-256 verified" on the step, and puts the hash that arrived in the state.

### Measured, against the live release

On this machine (GTX 1070, compute capability 6.1, CUDA 12.6 engine), a real update of the real
v0.1.40.1:

```
Download the engine         190.2 MB, SHA-256 verified
Inspect the archive         3 files, engine 0.1.40 for v0.1.40.1
Run the staged engine       it starts and prints its usage
Start it and check the version   running 0.1.40
```

18 s end to end. The digest came from the API as
`5ffaf2ba1fc68cc4cc7ef26889a0eaad5b302e251067a12f191911ea29770f42` and matched.

### Also fixed: the asset table had gone stale

It was a hardcoded map, and it did not know about `strata-windows-x64-cuda12.zip` (`setup.py`'s
`CUDA12_ASSET`, published since). A machine on the experimental CUDA 12 engine would have been handed the
CUDA 13 build, which it cannot run - and on this machine that is exactly what would have happened.

The name is now built from the platform plus **the CUDA major the installed engine's own `BUILD.json`
records**, so keeping an install on the CUDA line it already uses needs no code change when a new asset
appears, and an unknown CUDA falls back rather than refusing.

### Two bugs the live run found that no offline test had

**The tag is not the engine version.** `v0.1.40.1` is a hotfix release whose engine is `v0.1.40`, so the
exact comparison in the inspect step refused a perfectly good update. Only an *older* engine is a refusal
now - that means the wrong asset was picked. `_verify()` had the same comparison and the same bug: it
failed *after* a correct install and rolled back, which is the right behaviour for the wrong reason. The
panel repeated it too, saying "The engine is now v0.1.40.1" about a 0.1.40 engine.

### Please note: this is a new branch

`main` has moved ~1150 commits and its history was rewritten - the previous branch (#645) shared **no
ancestry** with upstream at all, so its diff was not reviewable. This is a fresh branch off `82f46a8` with
the work re-applied, not a rebase. #645 should be closed in favour of this.

### The UI port nearly broke the app, and nothing noticed

Splicing the panel into `app.js` with PowerShell line arrays silently dropped ~30 lines, including `fmt`,
`kfmt` and `gb`. `node --check` passed - the file still parsed. Same lesson as `vv0.1.38`: a syntax check
is not a functional check. The insert is now done in Python and verified by asserting that every top-level
name upstream defines still exists afterwards, and the app was re-checked in a browser. The same re-encode
turned the step marks into mojibake (`ľó`) in the browser; the block is now fetched as raw bytes and the
render check asserts the code points.

### Tests

```
python serve/test_update.py        110 checks
python serve/test_update_http.py   51 checks, real HTTP
python -m pytest serve             444 passed, 8 skipped
```

The hash cases are asserted as firmly as the rest: a matching digest updates and reports the hash; a
same-length file with different bytes is refused, deleted, and **no later step runs**; no digest is
refused; a non-sha256 digest is refused; a four-part tag with a three-part engine is accepted while an
older engine is refused; and the same refusal over HTTP.

The UI render check (61 checks) is **not** in the PR - it needs node and the repo has no JS harness.

### Still open from before, unchanged

`BUILD.json` with a BOM (`json.loads(read_text())` at `serve/server.py` raises on it, so an engine saved by
a Windows editor reports no version). One line, `encoding="utf-8-sig"`. I would rather send it as its own
PR than smuggle it in here. `get_prebuilt()` could use the same `digest` check described above - that is a
change in `setup.py` and I have not touched it.

En el sitio

Enlaces a install, modelos, releases.