#6265 [Engine] layer-split auto: the #2644 clamp picks a slower split on asymmetric same-VRAM pairs (measured +10% hand-pinned)
open · @cobra91 · 1 comentarios · En GitHub
BenchmarksMulti-GPUAMD / HIPNVIDIA / CUDAWindows
Descripción
**Hardware**: RTX 5080 (16 GB, PCIe x16 direct CPU) + RTX 4060 Ti (16 GB, PCIe x4 chipset) — equal VRAM, very different compute and bus. Ryzen 7 9800X3D, 32 GB RAM, Windows 11.
**Build**: engine 0.1.62 (v0.2.16), model Swift-Qwen3.8-Flash-Next GSQ-RCO IQ2_XS, spec 8 / spec-min-p 0.70 / mtp rt / kv q4_0 / kv-resident 32768 / pipeline-windows 2, gpu order 1,0.
The boot log shows the clamp discarding the faster-predicted placement:
```
layer split auto: K=2 would give a stage fewer than 3 layers (a tiny expert cache there clamps the prompt chunk of every card, #1616) and leave that card idle; chose K=20 instead, which gives each card at least its share of the layers by its room for experts (35.9 ms per decode window, not 24.2; --layer-split N sets it by hand, #2644)
```
Measured, same session, same recipe, only `--layer-split` differs (warm, greedy, 900-token counting prompt, 3 runs each):
- auto (K=20): 149.4 / 153.9 / 160.3 tok/s — median **153.9**
- `--layer-split 19`: 160.0 / 169.0 / 173.6 tok/s — median **169.0** (**+9.8%**)
On this class of rigs (equal VRAM, unequal compute/bus), #2644's balance rule ("each card at least its share of the layers") and the speed prediction disagree, and the clamp wins — leaving a measured ~10% (predicted 33%) on the table. Suggestion: when the clamped choice predicts clearly worse ms/window than an alternative, prefer the faster placement (or make the balance rule a preference rather than a hard clamp); #1616's tiny-cache concern can remain as a guard for that specific case.
Happy to test any change on this rig.En el sitio
Enlaces a install, modelos, releases.