Pull requests / #2627
#2627 native dense: count the STRATA_Q8_PACKED copies and allocate them after the required weights
open · @justincinmd · 0 コメント · GitHub で見る
AMD / HIPNVIDIA / CUDAModels & quants
本文
## Summary `STRATA_Q8_PACKED=1` (on by default on gfx1151) keeps a second, packed copy of each eligible Q8_0 dense matrix in VRAM. The layer split search did not count that copy. A copy that did not fit also aborted the load. This PR counts the copies and makes them optional. ## What changed - `NativeDense::weight_bytes_for` prices each packed copy with the same test that `load` uses. The split search no longer under-prices a stage with such matrices. - `NativeDense::load` makes every required allocation first (the matrices, the `STRATA_HC_Q8` copies, the q8_1 scratch) and the packed copies after them. Before, it made each copy right after its matrix. - The first copy that does not fit is dropped with one stderr line. The copies after it are not tried. A matrix without a copy uses the GGUF layout, as with the switch off. Any other failure is still an error. - `include/strata/core/optional_alloc.hpp` (new, header-only): the required-then-optional sequencer that `load` uses. - `tests/core/optional_alloc_test.cpp` (new, CPU only). Runs with Q8 packing off keep the same allocations in the same order. On gfx1151 the default path changes: a stage with eligible Q8_0 matrices is priced higher, so an automatic split can place layers differently, and a copy that does not fit no longer stops the load. ## Testing - `optional_alloc_test` drives the sequencer in `load`'s call order against a fake VRAM budget. Cases: 13 MiB free with three 4 MiB matrices (fails in the old order, loads now), room for some or all copies, a required allocation out of memory, a copy upload failure, and a required allocation after `finish()`. - HIP (ROCm 7.2.4, gfx1201) builds. CUDA 13.0.2 (sm_120) and oneAPI DPC++ 2026.1.1 (SYCL) build with no new diagnostics. Compile only. ## Extra Notes Not run on a GPU with an eligible matrix: no pack I have contains a Q8_0 dense matrix wide enough to pack. I found the problem while testing the gfx1151 decode switches on gfx12 (a separate PR). A gfx1151 run with a Q8_0 pack would confirm it.
関連リンク
インストール・モデル・リリースへの站内リンク。