#448 Layer split with a small second GPU caps prompt chunk at 512: 6.2x slower prompt reads, same decode

Description

### Environment

Engine 0.1.31, repo HEAD `9259cad` (2026-10-01). Measured 2026-10-01 on:

- RTX PRO 4500 Blackwell 32 GB (82 SM, CUDA arch 120) + RTX 3080 10 GB (68 SM, arch 86)
- Threadripper PRO 5975WX, 499 GB RAM, Linux
- model `qwen3.8-flash-next-ud-q4_k_xl` (Unsloth UD-Q4_K_XL split GGUF, 71.73 GiB routed experts
  loaded into the RAM arena), `--max-context 262144 --kv int8 --spec 4 --mtp --expert-cache auto`

---
### a small second GPU in a layer split collapses `prompt chunk auto` to 512 tokens (~6x slower prompt reads)

**Symptom.** Same model, same weights, same flags; the only difference is whether the 3080 takes a
slice of the layer split:

| arm | expert cache | `prompt chunk auto` | prompt read | decode |
| --- | --- | --- | --- | --- |
| pair (`--layer-split auto`) | CUDA0 8,566 slots + CUDA1 512 slots (1.27 GiB) | **512 tokens** | 127,688 tok in 295,684 ms = **431.8 tok/s** | 85.8 / 86.1 tok/s |
| single stage | CUDA0 7,927 slots | **8192 tokens** | 141,173 tok in 52,955 ms = **2,665.9 tok/s** | 84.8 tok/s |

Decode is unchanged (±1 %). Prompt reads differ by **6.2x**. On an agent workload — long prefixes,
modest generation, most requests re-reading 100-200K tokens — the pair arm is much worse overall,
so the small card's residency gain is not merely neutral, it hides a large prompt regression.

**Cause.** `src/program/generate.cpp:3663-3682`: the chunk is the largest candidate every
participant can fund, and a participant is each stage's own expert cache:

```cpp
auto fits = [&](int64_t c, bool cap) -> bool {
    for (const PfPart& p : pf_parts) {
        const int64_t k = part_slots(p, c);
        if (k <= 0 || k + 128 > p.cache->slots()) return false;
        if (cap && k * 100 > kAutoLendPct * p.cache->slots()) return false;
    }
    return true;
};
```

CUDA1's 512-slot cache must keep the 128-slot floor, so it can fund at most ~384 slots of prompt
staging — the binding constraint for every candidate above 512 tokens. CUDA0's 8,566 slots never
get a say. The comment above `fits()` states the assumption this exposes:

> the chunk is the largest one EVERY participant can lend (a smaller chunk only reads slower)

which is right for comparable cards and wrong when a participant is a small helper card.

Worth stating precisely, because it is easy to misdiagnose: an explicit `--prefill` is **not**
saved by the percentage cap (`fits(c, false)` — the auto scan alone applies `kAutoLendPct`, 85 % or
90 % depending on `Prefill::pinned_share()`). `--prefill 4096` on the pair arm halves down to 512
through the same slot floor and says so:

```
strata serve: prompt chunk 4096 -> 512 tokens so its buffers fit in every expert cache
```

So there is currently no operator setting that keeps a large chunk while the small card is in the
split — the only lever is taking it out of the split, which is a bigger change than tuning.

**Suggested fixes**, roughly in order of how little they disturb the current behaviour:

1. When the candidates are scanned, weight each participant by its share of the loan rather than
   demanding that the smallest cache fund the whole chunk — the large stage can carry most of the
   staging.
2. Or treat a participant whose cache is far below the others (say under 15 % of the largest) as
   unable to lend, letting it stream during the prompt instead of capping everyone.
3. At minimum, name the constraint when the fallback bites, e.g. `prompt chunk auto: 512 tokens
   (CUDA1's 512-slot expert cache cannot fund more)`, so a 6x prompt regression is traceable in one
   log line instead of needing an A/B against a single-card arm. `docs/SECOND_GPU.md` and
   `docs/DETAILS.md` describe split residency and prompt lending separately; neither notes that the
   two interact through this floor.

**Not a bug, for the record:** `--expert-cache-device1` on that same small card is the good use of
it — one stage plus a 1,800-slot tier left `prompt chunk auto: 8192` intact and raised decode from
84.8 to 91.1 tok/s at 167K context (hit rate 69.6 % -> 73.6 %). The tier's own cost is the
per-request synchronisation the docs warn about (`host 74 ms staging+launching, 124 ms waiting for
it`), and it grows with size. 1,800 vs 2,600 slots, requests of similar size:

| tier | hit rate | CUDA1 entries | active layer launches | wait |
| --- | --- | --- | --- | --- |
| 1,800 | 73.6 % | 6,428 | 2,926 | 124 ms (3.7 % of the request) |
| 2,600 | 78.0 % | 11,061 | 4,491 | 298 ms (6.2 %) |

Residency rose 4.4 points and decode did not (87.7 tok/s at 182K context vs 91.1 at 167K — the
contexts differ, so the wait column is the trustworthy comparison, and it moved 2.4x on 1.5x the
launches). Later, larger requests on the 2,600 tier waited 416-597 ms. The tier has a sweet spot
well below "as much as fits", which is worth a sentence in `docs/SECOND_GPU.md`.