#2596 layer-split auto: the split is not stable between starts, and a different K invalidates the whole on-disk tier (K=37 in 3 of 38 boots; 31 unreadable-chain lines)

Descrição

## What happens

`--layer-split auto` does not pick the same split on every start, and the planner says so itself — it prints a
near tie and names the alternative. On this pair it chose **K=38** on 35 of 38 starts and **K=37** on the other 3:

```text
strata generate: layer split auto: K=38 - predicted 27.9 ms per decode window; the caches hold 8185 of 24576 profiled pairs (~95.2% of the routed mass), best of 46 placements
strata generate: layer split auto: NEAR TIE - K=41 is predicted 28.6 ms per decode window, within 2.5% of K=38 (27.9 ms); the cards differ, so the prediction may not rank them (it prices decode, not the prompt). To try the other: --layer-split 41
```

```text
strata generate: layer split auto: K=37 - predicted 27.9 ms per decode window; the caches hold 8594 of 24576 profiled pairs (~95.4% of the routed mass), best of 46 placements
strata generate: layer split auto: NEAR TIE - K=40 is predicted 28.4 ms per decode window, within 2.0% of K=37 (27.9 ms); the cards differ, so the prediction may not rank them (it prices decode, not the prompt). To try the other: --layer-split 40
```

The split lands in the runtime identity, and the serve side shows the flip:

```text
strata serve: layer split: layers 0-37 (CUDA0), 38-47 (CUDA1), one hand-off per window
strata serve: layer split: layers 0-36 (CUDA0), 37-47 (CUDA1), one hand-off per window
```

When a start picks the other split, the state on disk written by the previous one is **unreadable**, and the tier
says it in its own words:

```text
strata serve: conversation cache: discard unreadable disk conversation (session file: saved with another layer range than this runtime's)
strata serve: segment tier: unreadable chain kept for the GC (session file: saved with another layer range than this runtime's)
```

## What we measured

- **38 starts** produced a chosen-split line in the log: **35 × K=38, 3 × K=37**. The three K=37 starts are
  consecutive log lines `K=37 … NEAR TIE - K=40` followed by `layer split: layers 0-36 (CUDA0), 37-47 (CUDA1)`.
- **Every one of the 38 starts printed `NEAR TIE`** (35 × "K=41 within 2.5% of K=38", 3 × "K=40 within 2.0% of
  K=37"). The margin the planner is deciding on is under 2.5% of a *predicted decode* cost, and the message admits
  the prediction does not price the prompt.
- **31** × `segment tier: unreadable chain kept for the GC (session file: saved with another layer range than this
  runtime's)`, plus **1** × `discard unreadable disk conversation` with the same reason. All 31 are in the window
  between the first K=37 start and the start after the last one — i.e. every K=38 start that followed a K=37 start
  could not read what it had written.
- Concrete cost, in a controlled run (same machine, same model, same 100k prompt): the **first** reload attempt
  after the split changed failed with exactly this message and fell back to a full prefill of 104.8 s; the **retry**
  with a stable split restored `99715` of `99722` tokens in 438.9 ms.

## Environment

- Windows 11 Pro (26200), WDDM. AMD Ryzen 9 9950X, 128 GiB RAM.
- 2 GPUs with an asymmetric link: RTX 4080 SUPER (PCIe 4.0 x16, probed 26.9 GB/s) + RTX 3090 in an x4 chipset
  slot (probed 1.0-1.7 GB/s across starts). Both cards differ in SM count and clock (80 SMs @2.55 GHz vs
  82 SMs @1.70 GHz), which is exactly the case the message says the model may not rank.
- Strata 0.1.44.1, source build with our local layer (conversation-cache disk tier with a `kMetaVersion=2` sidecar
  carrying the stage count, plus segment tier / restore ladder / archive).
- Flags: `--layer-split auto --batch 4 --max-context 196608 --kv int8 --kv-resident 32768 --conversation-cache-park
  disk --conversation-cache-disk-mib 460800 --conversation-cache-segments on`, model Qwen3.8-Flash-Next UnsLoth
  UD-Q4_K_XL (native pack).

## Why it matters

The disk tier's entire value is that the state survives a restart. `auto` is the default everyone runs, and it is
the one input that can invalidate all of it — not one entry: the sidecar carries the layer range, so 31 segment
chains were marked unreadable for the GC in the window above. A restart that is *supposed* to be free becomes a
full re-prefill of every parked conversation, and the user gets no warning that the previous start's choice was
arbitrary.

## What we propose

In order of cost:

1. **State the invariant in the log next to the tie.** The message already names the alternative; one more clause —
   "a different choice on the next start makes states saved with this one unreadable" — costs nothing and is
   exactly what a user needs in order to not use `auto` with a disk tier.
2. **Declare the split in the saved state and prefer it on the next start.** The identity is already recorded (the
   sidecar carries the stage count); if the runtime also recorded which split wrote it, `auto` could prefer that
   split when a state exists in the park directory, and the tier would stay valid across restarts.
3. **Pin it, as a documented mitigation.** `--layer-split <K>` works today and is what we run now. It is a
   workaround, not a fix: it costs the user the whole point of `auto`.
4. **Make the tie deterministic** (lowest K, or the split with fewer hand-offs) so at least the *same* machine picks
   the *same* split run to run. We have not verified why the choice flips; see below.

## Hypothesis (marked, not verified)

The tie-break is between two placements separated by under 2.5% of a predicted decode cost, and the prediction
admits it does not price the prompt. The inputs move between starts: the CUDA1 PCIe probe alone read 1.0, 1.2, 1.5
and 1.7 GB/s across the boots in this log, and the two cards differ in SM count and clock. We did not verify that
this probe variation is what flips K between 37 and 38 — it is the reading that best fits, not a finding.

## What we ruled out (checked before opening this)

Searched the tracker (open and closed) for `layer-split auto`, `NEAR TIE`, `"layer range"`, `stage count`,
`unreadable`, `split deterministic`, `split stable`:

- **#509** (closed/completed) `layer_split auto is 2.86x slower than a balanced K on prefill with a heterogeneous
  pair` — about *which* K `auto` picks for speed.
- **#1238** (closed/not planned) `auto … still picks K=2 on a P40 + RTX 3070` — also about which K, on old cards.
- **#2463** (open) `Layer-split auto (0.1.44.1): a 2x link gap is no longer priced, so a 4090 x16 + 3090 x8 pair
  picks K=45` — the *pricing* of the link, not the stability of the choice.
- **#747** (closed/completed) multi-conversation parking on multi-GPU layer-split — the feature whose state gets
  invalidated.
- **#2594** (open, 2026-10-15) a split reporting `0 block reads` — different subsystem (the KV streaming counter);
  cited only because it is the most recent split report and its rig has the same x16/x4 gap class.

None of these mentions run-to-run stability, and none mentions that a different `auto` choice invalidates the
on-disk tier.

## What we did not test

- We did not reproduce the flip on demand: the 3 K=37 starts happened in production, and we did not restart the
  engine 38 times to characterise the distribution. 35/3 is what the log contains, not a measured probability.
- We did not measure the prefill/decode difference between K=37 and K=38 on this pair (the planner's own prediction
  says they are within 2%).
- Windows only, one pair of cards, one model. Another rig (equal cards, or a symmetric link) may be perfectly
  stable; our point is that the default does not say what it costs when it is not.