Pull requests / #2287
#2287 setup: resume downloads when HEAD omits Content-Length
open · @KeyanHu-git · 0 comentários · No GitHub
Setup & installAMD / HIPWindows
Descrição
## Title setup: resume downloads when HEAD omits Content-Length Issue: Fixes #1915. ## Summary A clean early EOF can leave a partial model file under its final name when HEAD omits `Content-Length`. In #1915, setup renamed a 5.47 GB fragment of a 29.61 GB shard before the later size check rejected it. This change establishes the expected length before accepting completion. Short responses remain in `.part` and resume with Range; a transfer whose completeness cannot be established stays partial and reports why. ## What changed - **Expected length:** use ModelScope's published size when HEAD has none, and obtain the total from GET `Content-Range` or `Content-Length`. Fetch ModelScope metadata once and reuse it for the final SHA-256 check. - **Resume validation:** check the returned range and length before appending or clearing existing bytes. A valid full response to a Range request restarts from zero; a conflicting response preserves the partial file. - **Completion handling:** retain `IncompleteRead.partial`, resume interrupted chunked responses, and accept HTTP 416 as completion only when the reported total matches the local file and any known length. Complete chunked responses and explicit empty files remain supported. - **Regression tests:** add `tools/test_setup_download_length.py` with local HTTP fixtures for early EOF, ModelScope metadata reuse, Range handling, HTTP 416 and completion boundaries. ## Extra Notes ### Validation Windows 11, Python 3.12.0; base: `a0116b2`. | Check | Result | | --- | --- | | Two root-cause regressions on unmodified upstream | One assertion failure and one error | | Same regressions with this patch | Both pass | | Download-length and ModelScope suites | **23 tests pass**, including 14 new tests | | Full setup suite on upstream | 474 tests; one existing SYCL assertion failure | | Full setup suite with this patch | 488 tests; the same SYCL assertion failure | ```bash python -m unittest tools.test_setup_download_length tools.test_setup_modelscope python -m unittest discover -s tools -p 'test_setup_*.py' ``` The root-cause fixtures check resumed offsets, final bytes and hash verification. The unchanged full-suite failure is `test_setup_sycl.ToSycl.test_xe_ranks_the_experts_with_the_sycl_profile_and_i915_keeps_the_shipped_one`. A separate bounded probe of the real ModelScope URL on this host found no length in HEAD. GET advertised 65,536 bytes and a total of 29,608,446,496 bytes, but returned EOF after 47,824 bytes. This checks the real transport condition; the complete model was not downloaded again. ### Compatibility and related work The download interface is unchanged. Close-delimited responses without a known length now stop with their `.part` retained; a completion mark without a published hash still records transfer completion rather than cryptographic integrity. #1405 also refactors `download()` and local-file reuse. Its transfer loop still obtains the length from HEAD. If it lands first, this patch will need to be integrated with that refactor.
No site
Links install, modelos, releases.