Pull requests / #2627

#2627 native dense: count the STRATA_Q8_PACKED copies and allocate them after the required weights

open · @justincinmd · 0 comments · View on GitHub

AMD / HIPNVIDIA / CUDAModels & quants

Description

## Summary

`STRATA_Q8_PACKED=1` (on by default on gfx1151) keeps a second, packed copy of each eligible Q8_0 dense matrix in VRAM. The layer split search did not count that copy. A copy that did not fit also aborted the load. This PR counts the copies and makes them optional.

## What changed

- `NativeDense::weight_bytes_for` prices each packed copy with the same test that `load` uses. The split search no longer under-prices a stage with such matrices.
- `NativeDense::load` makes every required allocation first (the matrices, the `STRATA_HC_Q8` copies, the q8_1 scratch) and the packed copies after them. Before, it made each copy right after its matrix.
- The first copy that does not fit is dropped with one stderr line. The copies after it are not tried. A matrix without a copy uses the GGUF layout, as with the switch off. Any other failure is still an error.
- `include/strata/core/optional_alloc.hpp` (new, header-only): the required-then-optional sequencer that `load` uses.
- `tests/core/optional_alloc_test.cpp` (new, CPU only).

Runs with Q8 packing off keep the same allocations in the same order. On gfx1151 the default path changes: a stage with eligible Q8_0 matrices is priced higher, so an automatic split can place layers differently, and a copy that does not fit no longer stops the load.

## Testing

- `optional_alloc_test` drives the sequencer in `load`'s call order against a fake VRAM budget. Cases: 13 MiB free with three 4 MiB matrices (fails in the old order, loads now), room for some or all copies, a required allocation out of memory, a copy upload failure, and a required allocation after `finish()`.
- HIP (ROCm 7.2.4, gfx1201) builds. CUDA 13.0.2 (sm_120) and oneAPI DPC++ 2026.1.1 (SYCL) build with no new diagnostics. Compile only.

## Extra Notes

Not run on a GPU with an eligible matrix: no pack I have contains a Q8_0 dense matrix wide enough to pack. I found the problem while testing the gfx1151 decode switches on gfx12 (a separate PR). A gfx1151 run with a Q8_0 pack would confirm it.

Related on this site

Quick links to related pages.