#2644 layer-split auto gives an asymmetric pair 3 of 48 layers (12 GB idle, -44% prefill)

Descrição

## What happens

On an **asymmetric** two-GPU pair, `layer-split auto` assigns almost all layers to one card, leaving most of the second card's VRAM unused for the whole run.

Measured on RTX 4060 Ti 16 GB + RTX 3060 12 GB (48 layers, Strata 0.1.44.1, `296b59d2`, Flash-Next IQ3_S):

```
layer split, CUDA1: 5.50 GiB free of 11.63, room for experts 4.72 GiB
layer split: CUDA1 runs layers 18-47 ...     (with a manual split)
```

With `auto`, the engine log reports:

```
layer split auto: K=2 would give a stage fewer than 3 layers (a tiny expert cache
  there clamps the prompt chunk of every card, #1616); chose K=3 instead
layer split: CUDA1 runs layers 3-47, expert cache 4263 slots
layer split: CUDA0 runs layers 0-2
```

So the RTX 3060 got **3 of 48 layers**. Through an entire 14,564-token prefill it drew **17 W** (idle power) and sat at ~0-1 % utilisation while the 4060 Ti ran at 100 %:

```
samples (1/s) during prefill, host CPU busy% | GPU0 util,mem,power | GPU1 util,mem,power
 8  cpu_busy=9%  gpus=[0, 100, 11614, 50.58; 1, 100, 15555, 102.25;]
10  cpu_busy=19% gpus=[0, 100, 11614, 135.79; 1,   8, 15559, 100.07;]
12  cpu_busy=16% gpus=[0, 100, 11614, 138.12; 1,   0, 15730,  31.58;]
```

Host CPU was ~3 % and cross-card expert traffic measured 0.0-4.1 % of routed experts, so neither the CPU nor the PCIe link was the limit — the split itself was.

## Cost

Identical cold 14,564-token prompt, fresh server per point, only `layer_split` changed:

| `layer_split` | RTX 3060 layers | RTX 4060 Ti layers | prefill tok/s |
| ---: | ---: | ---: | ---: |
| `auto` | 3 | 45 | 452.6 |
| 12 | 12 | 36 | 581.8 |
| 18 | 18 | 30 | 693.0 |
| 24 | 24 | 24 | 761.7 |
| **28** | **28** | **20** | **802.1** |
| 32 | 32 | 16 | 803.2 |

**`auto` costs 44 % of prefill throughput on this pair** versus split 28, where the curve plateaus. Decode improves too (41.8 → 50.4 tok/s chat median; 25.1 → 36.2 agent median), presumably because fewer layers per card lets each expert cache cover its own layers better.

## A diagnostic that may be worth documenting

With the mis-set split, prefill is **flat in prompt length** — 491.7 tok/s at 4K, 512.1 at 16K, no batching benefit. With split 28 it becomes a slope: 488.2 at 4K, 835.0 at 16K. Note the 4K number is unchanged, so **a 4K-only benchmark cannot see this**. Anyone tuning from a short prompt would conclude the install is fine.

## Which way the split should go is not intuitive

The obvious rule — give the faster card proportionally more layers — predicts the wrong direction here. What tracked the result was **memory bandwidth**, not SM count or FP32 TFLOPS: the RTX 3060 has 360 GB/s against the 4060 Ti's 288 GB/s, so it is the better host for expert-heavy layers despite fewer and slower SMs. That is an argument for a bandwidth-aware automatic split rather than a compute-aware one.

## Reproduction

```
RTX 4060 Ti 16 GB (CUDA1) + RTX 3060 12 GB (CUDA0), 48 layers
Strata 0.1.44.1, original Flash-Next IQ3_S, --max-context 262144 --kv int8
strata-iq3_s.json:  "gpu": [1, 0],  "layer_split": "auto"
```

Then set `"layer_split"` to `12` / `18` / `24` / `28` / `32` and cold-prefill one fixed long prompt against a freshly started server at each value.

## Related

Full benchmark and the sweep are in PR #2643, which adds the report under `bench/results/2026-10-15-community-rtx-4060-ti-plus-rtx-3060-qwen3-8-flash-next-iq3-s-3/`.

## Not claiming

The `auto` heuristic avoids a real failure mode (#1616 — a stage whose tiny expert cache clamps the prompt chunk of every card). I did not observe that clamping at any split I tested, but I only tested this one pair, on one model, at one context length. If a bandwidth-aware balance would reintroduce #1616 on other hardware, the tradeoff may well be deliberate — in which case documenting "sweep `layer_split` on asymmetric pairs" would be enough.