改动 / #3559

#3559 [Engine] native_dense: STRATA_HC_Q8 uploads only the stage's own Q8_0 rows on a trimmed layer split

open · @havok2-htwo · 0 评论 · 去 GitHub 看

Setup & installMulti-GPUAMD / HIPNVIDIA / CUDAModels & quants

说明

[Engine] native_dense: STRATA_HC_Q8 uploads only the stage's own Q8_0 rows on a trimmed layer split

## Problem
With `--trim-stage-weights` and `STRATA_HC_Q8=1`, the loop that attaches the GGUF's Q8_0 hyper-connection rows does not skip other stages' layers. Each trimmed card uploads all 48 layers' rows and reads only its own. Observed: 0.63 GiB per card with UD-IQ4_XS.

## Change
`src/core/native_dense.cpp`, +2 lines: the loop skips a tensor when `outside(name) || !in_range(name)`, the filter the upload loop above it already uses. No value changes; without stage trim `in_range` passes every layer, so nothing changes.

## Validation
- Revision: v0.2.0-beta `3d1fb480` plus this commit (`023b3962`), CUDA sm_86 build with setup-style flags, 4x RTX 3090 24 GB.
- Rig: Qwen3.8-Flash-Next UD-IQ4_XS native GGUF, `--layer-split 12,25,37 --trim-stage-weights`, `STRATA_HC_Q8=1`, 16 slots, greedy.
- Per-card `STRATA_HC_Q8=1: ... GiB` line: stock 0.63 / 0.63 / 0.63 / 0.63; with this change 0.16 / 0.18 / 0.16 / 0.15.
- Tokens (`tools/batch_test.py --batch 16 --n 16 --max-new 128`, same expert cache on every card): solo 16/16 and batch 16/16 identical between stock and this change.
- Without trim, 0.63 GiB per card with and without the change, as expected.

## Not tested
HIP, Metal and SYCL builds (this file is compiled there, but not built here), 2- and 3-GPU splits, `--layer-split auto`, long prompts, other models.

本站相关内容

相关页面的快捷入口。