#6149 [Engine] generate: skip post-search CUDA0 reload when stage_trim already trimmed (#6110)

closed · @wronglebowsk · 0 Kommentare · Auf GitHub

BenchmarksMulti-GPUAMD / HIPNVIDIA / CUDAModels & quantsLinux

Beschreibung

## Problem and result

On an explicit `--layer-split` with `--trim-stage-weights` (or when the early spill walk converts `--layer-split auto` into an explicit split), `stage_trim` (`src/program/generate.cpp:5580`) already trims `CUDA0` before its initial `wt.load` and `native_dense.load` (`src/program/generate.cpp:5607-5612`).

However, `const bool trim = o.trim_stage_weights && multi_gpu && !stages.empty();` at `src/program/generate.cpp:6548` (used for `--layer-split auto` post-search trimming) did not check `!stage_trim`. As a result, on an explicit split:

1. `if (trim)` at line 6551 scanned `index.txt` and ran `WeightTable::pool_bytes` 48 times over the already-trimmed `skip` set.
2. `if (trim)` at line 7168 freed `CUDA0`'s already-trimmed weight arena and `NativeDense` projections, re-allocated the same size, and reloaded all 191 native projection matrices (`1,146.24 MiB`) from the GGUF shards a second time, adding ~5-7 seconds to startup and logging `CUDA0 0-17 (0 MiB freed)`.

With `&& !stage_trim` added, `CUDA0` loads its trimmed weights once at startup and skips the redundant `index.txt` scan and reload.

## Related discussion

Fixes https://github.com/Niko1221/Strata/issues/6110

## Approach

Add `&& !stage_trim` to `const bool trim` at `src/program/generate.cpp:6548`. When `stage_trim` is false (`--layer-split auto` with stage weight trimming), behavior is unchanged. Single-line in-place edit so `data/settings_register.json` line numbers do not shift.

## Validation

**Passed:**
- `python3 tools/test_settings_register.py && python3 tools/settings_register.py --check` on commit `c63f6de68` (`Ran 8 tests in 0.943s OK`).
- Built `strata` (CUDA 13.4, `-DSTRATA_ENABLE_CUDA=ON -DSTRATA_MMQ=ON`) and verified startup with `--pack /home/paul/Strata-data/packs/unsloth_iq3_xxs --native .../Qwen3.8-Flash-Next-UD-IQ3_XXS-00001-of-00004.gguf --ple-gguf .../Qwen3.8-Flash-Next-UD-IQ3_XXS-00004-of-00004.gguf --layer-split 18,33 --trim-stage-weights` on 3x NVIDIA GPUs (`RTX 5060 Ti 16 GB + RTX 2080 Ti 11 GB + RTX 3080 Ti 12 GB`, `2x Xeon E5-2683 v4`, `188 GiB RAM`, Ubuntu Linux 7.3.4):
  - Before: `CUDA0` loaded `191 native projection matrices, 1146.24 MiB of weights` twice (`5.1 s` + `5.0 s`) and printed `strata generate: layer split: CUDA0 keeps only its own layers' dense weights (0.43 GiB in all): CUDA0 0-17 (0 MiB freed)`.
  - After: `CUDA0` loads its trimmed weights (`layers 0-17 only`, `95 MiB` canonical + `1146.24 MiB` native dense) once in `5.0 s`, and `tools/strata_bench.py --long` + `strata.py bench run --suite standard-v1` (21/21 requests, 12/12 sanity checks) pass with identical VRAM allocation (`10,575` resident GPU expert slots).

**Failed:**
- none observed

**Not run:**
- HIP, SYCL, and Metal builds (`sycl/src/program/generate.cpp` and `metal/` do not have the `trim` post-search reload block).
- `--layer-split auto` hardware re-test (kept on the existing `stage_trim == false` path).

## Limits

- Only affects explicit `--layer-split` (or early spill-walk splits where `stage_trim` is true); `--layer-split auto` still loads `CUDA0` before the split search and trims it afterwards.

Mehr auf der Site

Links zu Install, Modellen, Releases.