#6110 [Engine] explicit --layer-split reloads CUDA0 dense weights a second time at startup (0 MiB freed)

closed · @wronglebowsk · 1 comentarios · En GitHub

Setup & installServer & APIMulti-GPUNVIDIA / CUDAModels & quantsWindowsLinux

Descripción

### Area

[Engine]  (speed, kernels, models, the server and API)

### Desktop app, Terminal or source?

Source (START-HERE.bat, setup.sh or a build)

### What happened

On an explicit `--layer-split` (for example `--layer-split 18,33 --trim-stage-weights`, or when the early spill walk converts `auto` into an explicit split), `stage_trim` (`src/program/generate.cpp:5580`) already trims `CUDA0` before its initial `wt.load` and `native_dense.load` (`src/program/generate.cpp:5607-5612`):

```text
strata generate: layer split: CUDA0 loads the dense weights of layers 0-17 only
strata generate: 95 MiB of weights loaded from .../packs/unsloth_iq3_xxs in 0.2 s (855 canonical tensors skipped: served natively)
strata: STRATA_HC_Q8=1: 0.24 GiB of Q8_0 hyper-connection projections for the verify read
strata generate: 191 native projection matrices, 1146.24 MiB of weights, in 5.1 s
```

However, `const bool trim = o.trim_stage_weights && multi_gpu && !stages.empty();` at `src/program/generate.cpp:6554` (added for `--layer-split auto` post-search trimming) does not check `!stage_trim`. As a result, on an explicit split:

1. `if (trim)` at `src/program/generate.cpp:6557` scans `index.txt` and runs `WeightTable::pool_bytes` 48 times over the already-trimmed `skip` set.
2. `if (trim)` at `src/program/generate.cpp:7168` frees `CUDA0`'s already-trimmed weight arena and `NativeDense` projections, re-allocates the same size, and reloads all 191 native projection matrices (1,146.24 MiB) from the GGUF shards a second time, adding ~5-7 seconds to startup and logging `0 MiB freed`:

```text
strata: STRATA_HC_Q8=1: 0.24 GiB of Q8_0 hyper-connection projections for the verify read
strata generate: 191 native projection matrices, 1146.24 MiB of weights, in 5.0 s
strata generate: layer split: CUDA0 keeps only its own layers' dense weights (0.43 GiB in all): CUDA0 0-17 (0 MiB freed) (the later cards trim in late_weights)
```

Expected behavior: when `stage_trim` is already true (`const bool trim = o.trim_stage_weights && multi_gpu && !stages.empty() && !stage_trim;`), skip the redundant post-search `trim_prefix` scan and `CUDA0` reload.

### Smallest steps or command that reproduce it

```bash
./build/strata --serve \
  --pack /path/to/packs/unsloth_iq3_xxs \
  --native /path/to/Qwen3.8-Flash-Next-UD-IQ3_XXS-00001-of-00004.gguf \
  --ple-gguf /path/to/Qwen3.8-Flash-Next-UD-IQ3_XXS-00004-of-00004.gguf \
  --layer-split 18,33 --trim-stage-weights
```

### Reproduction status

Reproduced locally

### Strata version

Strata 0.2.15 · engine 0.1.61 (commit c77572806)

### Graphics card, memory and system

RTX 5060 Ti 16 GB + RTX 2080 Ti 11 GB + RTX 3080 Ti 12 GB · 2x Xeon E5-2683 v4 · 188 GiB DDR4 RAM · Ubuntu Linux 7.3.4

### Model, size and settings

Qwen3.8-Flash-Next UD-IQ3_XXS · 512K context (--kv k8v4) · --layer-split 18,33 --trim-stage-weights

### Logs

```text
strata generate: layer split: CUDA0 loads the dense weights of layers 0-17 only
strata generate: 95 MiB of weights loaded from .../packs/unsloth_iq3_xxs in 0.2 s (855 canonical tensors skipped: served natively)
strata: STRATA_HC_Q8=1: 0.24 GiB of Q8_0 hyper-connection projections for the verify read
strata generate: 191 native projection matrices, 1146.24 MiB of weights, in 5.1 s
...
strata: STRATA_HC_Q8=1: 0.24 GiB of Q8_0 hyper-connection projections for the verify read
strata generate: 191 native projection matrices, 1146.24 MiB of weights, in 5.0 s
strata generate: layer split: CUDA0 keeps only its own layers' dense weights (0.43 GiB in all): CUDA0 0-17 (0 MiB freed) (the later cards trim in late_weights)
```

En el sitio

Enlaces a install, modelos, releases.