#4522 [Engine] layer split: trim every card's dense weights on --layer-split auto too (from Strata-DualGPU, #642)
closed · merged 2026-11-15 · @cobra91 · 0 commentaires · Sur GitHub
BenchmarksServer & APIMulti-GPUAMD / HIPNVIDIA / CUDASecurityDocumentationWindows
Description
## Problem and result
Main already trims each card's dense weights to its own layer range, but only with an explicit `--layer-split`, opt-in, off by default - so the common path (`--layer-split auto`, what the app and most setups use) never trims, and CUDA0 never trims at all. Result measured on a 4060 Ti + 5080 / 32 GB rig, Swift 1.5 IQ2_XS, 64K, q4_0 KV: CUDA0's expert cache holds 6812 slots and decode runs at 33.9-38.8 tok/s. With this change every card, including the first, reloads only the dense weights of the layers it runs and the freed VRAM goes to its expert cache: 8021 slots on CUDA0 (+19%) and decode 52.0 tok/s - **+34 to +53%**, `--no-trim-stage-weights` restoring today's behaviour.
## Related discussion
The implementation is Hardin22's per-card trimming (Strata-DualGPU `d9ff514`, part of what #642 described); JeanP00l's #639 explored the same idea for explicit splits. Our measurements of the fork on this GPU class: #4470. No maintainer agreement on the default yet - flipping it to on-by-default is the one scope point worth a call, and I am happy to keep it opt-in if preferred.
## Approach
`NativeDense::release()` / `bytes_in()` / per-layer byte accounting, alongside main's layer-range `load` and `set_layer_range`; the freed bytes feed the split search (`trim_frees()` added to `stage_left()`); after the first card's trimmed reload the six `blk.1.ple_*` weights are wired again (`wire_ple`), which is also what makes a reload safe with the PLE; `--trim-stage-weights` / `--no-trim-stage-weights` flags. Later stages keep loading through main's existing `late_weights` path - this PR only adds CUDA0's trimmed reload, which main never does. The second commit fixes a plain bug of the reload: an explicit split loads its later stages eagerly and the `set_layer_range` they leave behind intersected with CUDA0's own range to nothing ("no supported GDN/QSA matrices"); each reload now resets the range to its own stage first.
## Validation
Engine built from main at 46c152a1 (MSVC 19.51 / Ninja / CUDA 13.4, arch 89;120), with and without this PR, same host (`serve/server.py`), same prompts, greedy. Rig: RTX 5080 + RTX 4060 Ti 16 GB, Ryzen 7 9800X3D, 32 GB RAM, Windows 11.
**Passed:**
- Auto split load: `layer split: each card keeps only its own layers' dense weights (2.86 GiB in all): CUDA0 0-22 (1497 MiB freed), CUDA1 23-47 (1431 MiB freed)`; expert cache CUDA0 6812 -> 8021 slots.
- Decode, first generation after load: 38.8 -> 52.0 tok/s; warm: 33.9 -> 52.0.
- Prefill, a 49,361-token prompt (same rig, greedy, engine-log rate): **228 -> 1,181 tok/s (5.2x)**. Without the trim the expert caches hold ~6,800 slots and the prefill streams most expert blobs from the SSD; with it the caches grow to ~8,200 + 7,500 and prefill reads from VRAM. (The full fork reads the same prompt at 2,310-2,506 tok/s; the remaining gap is the layer-major super-chunk engagement, a separate change.)
- Explicit `--layer-split 19`: loads, trims, and runs (after the second commit; it crashed with "native dense: already loaded"-class state before it).
- Output equality: not re-measured here; the fork's author compared this path over 5,333 teacher-forced tokens (perplexity 7.28 vs 7.24, same top token at 93.9% of positions, his docs/DUAL_GPU.md), which matches main's own note that expert placement already rounds differently.
**Failed:**
- none observed after the second commit.
**Not run:**
- HIP, SYCL and Metal builds (no hardware); the desktop app's own host; non-Qwen models (the reload path is model-agnostic but only Flash-Next family packs were available).
## Limits
Measured on one rig and one model family. The freed VRAM lands in the expert caches, so a placement that counted it can fill the cards harder (with #4534's search price the pair lands low on free VRAM on the later card; `--vram-reserve-later-mib` covers the margin, and the fork has two follow-up changes for that which I can bring separately).
Sur le site
Liens install, modèles, releases.