Pull requests / #2385

#2385 planner: rank auto layer splits for prompt pipelines

open · @MarsLuay · 0 comments · View on GitHub

BenchmarksSetup & installMulti-GPUNVIDIA / CUDAModels & quants

Description

## Summary

- Add `--layer-split-objective interactive|prompt|decode` for `--layer-split auto`.
- Make the default `interactive` objective first select the legacy serial-decode optimum, then optimize the slowest prompt-pipeline stage only among placements whose predicted serial decode cost does not regress.
- Keep `prompt` as an explicit unconstrained prompt-throughput priority and `decode` as the serial stage-sum ranking.
- Preserve explicit layer splits unchanged and apply the same objective to four-stage candidate scoring.

This addresses the heterogeneous-GPU planner mismatch where maximizing cache coverage or prompt throughput can select a decode-slow split. The planner remains model-based and does not benchmark at startup. The decode guard is a prediction constraint, not a universal hardware guarantee.

Related: #2052, #509, #2085

## Measured evidence

These are actual same-host measurements from 2026-10-14 using Strata **0.1.40** (version-scoped evidence, not a claim about current 0.1.44 throughput): RTX 3080 Ti + Tesla V100, SC117 IQ3_S, 262,144 context, 259,841-token prompt, `int8`, 32K KV resident, speculative decoding 4, and no prompt reuse.

| planner/config | prompt time | prompt rate | decode | cache hit |
|---|---:|---:|---:|---:|
| **Before:** `--layer-split auto` | 228.6 s | 1,136.7 tok/s | 31.4 tok/s | 90.9% |
| **Reference:** explicit balanced `--layer-split 24` | **169.2 s** | **1,535.4 tok/s** | **28.7 tok/s** | 78.1% |

The balanced split reduced prompt time by 26.0% and improved prompt throughput by 35.1%, despite its lower cache-hit percentage. These measurements motivate the prompt objective; they are not post-PR binary results.

A fresh 128K request was also run against the already-running **0.1.40** runtime with explicit balanced `--layer-split 24`: 124,631 prompt tokens, 76.91 s prompt time, 1,620.5 prompt tok/s, 7.86 s decode, 32.6 decode tok/s, 74.5% cache hit, no prompt reuse. This is version-scoped runtime remeasurement only; it does not validate this unbuilt PR head. The current CUDA toolchain could not produce a post-PR GPU binary.

## Validation

- `cmake --build build-pr --config Debug --target split_rules_test`
- `ctest --test-dir build-pr -C Debug --output-on-failure -R '^(split_rules_test|share_rules_test|host_limits_test|arch_defaults_test|batch_read_test|batch_groups_test|split_ring_test|draft_kv_plan_test|coupled_draft_test|controller_test|message_boundary_test|conv_cache_test)$'` — 12/12 passed
- `git diff --check`

A full host build remains blocked by existing unrelated AVX1 linker errors in the native expert test targets. A CUDA build was attempted with the installed CUDA 11.8 toolchain: CMake reached the changed planner dependencies, but the source build remains blocked by CUDA 11.8's missing C++20 support and the existing `cudaLaunchKernelEx` failure in `src/kernels/cuda/sampler.cu`.

Related on strata.com

Editorial links to help you install, pick models, or read release notes — not part of the upstream thread.