Pull requests / #2427

#2427 HC_Q8: skip the layers' bf16 hyper-connection copies on a layer split too

open · @paltian · 0 comentarios · En GitHub

Multi-GPUAMD / HIPNVIDIA / CUDAModels & quantsDocumentation

Descripción

## Title
Issue: none found for this. It extends the single-GPU HC_Q8 path (b60aab1b) to a layer split; #704 was the requantization approach and was closed as "not with a layer split".

## Summary
`STRATA_HC_Q8=1` leaves the layers' bf16 hyper-connection down/up projections out of the arena, but `add_hc_skip` was gated on `!multi_gpu` (`generate.cpp:3064`). On a layer split the bf16 copies stayed in the arena and the Q8_0 copies were added on top, so HC_Q8 did not free VRAM there — it cost ~0.63 GiB/card instead (see the "single GPU" note in b60aab1b and the layer-split comment in `native_dense.cpp:413`).

## What changed
`src/program/generate.cpp`, one gate: drop `!multi_gpu` from the `add_hc_skip` condition.

The prompt path (`prefill.cpp:2611`) and the token read (`layer.cpp:1410`) already take these rows through `hc_bf16_view`, which dequantizes the Q8_0 into a per-device scratch, so a layer split can drop the bf16 copies exactly as the single-GPU path does. `add_hc_skip` adds all `hc_*_down/up` names; each stage's loader still limits itself to its own layers through `in_range` / `add_foreign`.

## Extra Notes
Measured on UD-Q4_K_XL, two RTX 5060 Ti (sm_120), `--layer-split auto`, `STRATA_HC_Q8=1`, otherwise identical config:

| | before | after |
|---|---:|---:|
| canonical tensors skipped | 303 | 495 (+192 = the 4x48 `hc_*_down/up` rows) |
| dense weights loaded from the pack | 1416 MiB | 216 MiB |
| CUDA1 expert cache | 2143 slots | 2538 slots |
| split search | K=5, 4799 profiled pairs | K=7, 6083 profiled pairs |

A short request answers correctly (German capitals question -> "Paris, Rom", coherent reasoning, no errors). The read only moves from the bf16 copy to the Q8_0 copy; the single-GPU KL of that read is in `docs/QUANTS.md` (UD-Q4_K_XL: mean 0.0562 -> 0.0526, p99 0.925 -> 0.716). The kernels are untouched, so `gr_parity` is unaffected.

Not tested: HIP (the code path is shared; gfx12 defaults HC_Q8 on), a split with `--keep-canonical` (the gate sits inside `!keep_canonical`, unchanged), and batch serving.

En el sitio

Enlaces a install, modelos, releases.