Issues / #2393
#2393 Improve heterogeneous-GPU --layer-split auto selection for prompt and decode
open · @MarsLuay · 0 Kommentare · Auf GitHub
BenchmarksMulti-GPUNVIDIA / CUDAWindows
Beschreibung
## Problem On heterogeneous multi-GPU systems, `--layer-split auto` can choose a placement that looks good for one phase of inference but is poor for another. Long prompt processing is pipelined across stages, so its throughput is dominated by the slowest stage. Decode windows, however, visit stages serially, so their cost is closer to the sum of the stage costs. The previous automatic ranking did not make that tradeoff explicit. A prompt-oriented choice can improve prefill while regressing interactive decode, while a decode-only choice can leave substantial prompt throughput on the table. Cache-hit percentage alone is also not a sufficient placement objective. The problem is visible on mixed cards such as an RTX 3080 Ti plus Tesla V100. In version-scoped Strata 0.1.40 measurements, the automatic split took 228.6 seconds for a 259,841-token prompt (1,136.7 tok/s), while an explicit balanced `--layer-split 24` took 169.2 seconds (1,535.4 tok/s). Those measurements motivated improving the planner rather than requiring users to hand-tune split points. ## Resolution Implemented by [PR #2385](https://github.com/Niko1221/Strata/pull/2385): - Make `interactive` the default automatic objective. - First find the legacy serial-decode optimum. - Optimize the prompt pipeline only among placements whose predicted decode cost does not regress. - Retain explicit `prompt` and `decode` objectives for users who want a specific priority. - Leave explicit layer splits unchanged. The guard is model-based and prevents predicted regressions; it is not a universal hardware-performance guarantee. ## Related work - [PR #2385](https://github.com/Niko1221/Strata/pull/2385) — implementation - [#2052](https://github.com/Niko1221/Strata/issues/2052) — related heterogeneous-GPU layer-split behavior - [#509](https://github.com/Niko1221/Strata/issues/509) — related automatic split/cache-placement behavior - [#2085](https://github.com/Niko1221/Strata/issues/2085) — related multi-GPU planning behavior
Mehr auf der Site
Links zu Install, Modellen, Releases.