#2595 serve: on a layer split the conversation cache discards a whole image for "stage count: saved=0 current=1" (40 in one boot; one of 165,543 tokens)

Descripción

## What happens

On a 2-GPU layer split the conversation cache keeps discarding the on-disk image and falling back to a full
re-prefill. The engine prints both numbers, and the image it is holding carries **zero** stage parts while this
runtime expects **one**:

```text
strata serve: segment tier: hit 165543 tokens (whole live)
strata serve: conversation cache: discard snapshot (stage count: saved=0 current=1, 165543 tokens)
```

The request right after that discard re-reads the entire conversation:

```text
strata serve: prompt 171130 tokens = 0 reused + 171130 read in 234999 ms (728.2 tok/s), 1 generated in 89 ms (11.3 tok/s), drafts accepted 0 of 0, 6 checkpoints
```

So 235 s of prefill where a restore on the same machine and model costs well under a second
(`restored 171516 tokens (disk/checkpoint) in 656.9 ms`). The image that was thrown away was 165,543 tokens: not a
stale fragment, a whole conversation.

The small ones show the same words, and every one of them is preceded by a segment-tier line — the tier hands
something back, the restore validates it, and it is discarded:

```text
strata serve: segment tier: miss (the prompt ends before this segment, matched=1 tokens)
strata serve: segment tier: hit 1 tokens
strata serve: conversation cache: discard snapshot (stage count: saved=0 current=1, 1 tokens)
```

## What we measured

One production boot on this machine (`[strata] 2026-10-14 23:20:24 engine started`), read on 2026-10-15 while the
log was still growing. The count is a moving target; every reading is given:

- `discard snapshot (stage count: saved=0 current=1, …)`: **40** in that boot (28 at 01:20, 36 at 01:58, 39 at
  02:31, 40 at 02:35).
- Size histogram of those 40: **39 are `1 tokens`**, and **exactly one is above 1,000: `165543`**.
- The 165,543-token one is immediately preceded by `segment tier: hit 165543 tokens (whole live)`; the 1-token ones
  by `segment tier: hit 1 tokens` (39) or `segment tier: miss (… matched=1 tokens)` (44 + 5 over the same boot).
- Before that, in a 49.5-minute window on 2026-10-14, the same signature appeared 8 times with 3 images between
  100k and 133k tokens.

The counter is not the problem and the discard is not cosmetic: `saved=0` is the engine's own statement that the
image it holds has no stage parts, and the request that follows pays the whole prompt again.

## Environment

- Windows 11 Pro (26200), WDDM. AMD Ryzen 9 9950X, 128 GiB RAM.
- 2 GPUs: RTX 4080 SUPER (PCIe 4.0 x16) + RTX 3090 in an x4 chipset slot. Probed links in the log: 26.9 GB/s on
  CUDA0 vs 1.0-1.7 GB/s on CUDA1.
- Strata 0.1.44.1, source build with our local layer (conversation-cache disk tier + segment tier + restore ladder
  + archive). `strata.exe --serve` with:

```text
--pack …/unsloth-ud-q4_k_xl --native …/Qwen3.8-Flash-Next-UD-Q4_K_XL-00001-of-00004.gguf
--expert-profile …/expert-profile.bin --expert-cache auto --prefill auto
--spec 4 --mtp …/mtp/rt --spec-min-p 0.5 --pool-workers 10
--max-context 196608 --kv int8 --kv-resident 32768 --vision --vram-reserve-mib 700
--layer-split auto --batch 4 --tail-role-token 8678
--conversation-cache-mib 8192 --conversation-cache-slots 4
--conversation-cache-park disk --conversation-cache-park-min-free-mib 512
--conversation-cache-spill-dir D:\…\slots\xl --conversation-cache-disk-mib 460800
--conversation-cache-spill-when-full evict-oldest --conversation-cache-segments on
--conversation-cache-segment-budget-mib 65536
--conversation-cache-archive-mode state --conversation-cache-archive-keep 4
```

- Model: Qwen3.8-Flash-Next UnsLoth UD-Q4_K_XL (native pack, 24576 experts, shipped expert profile).

## What I think is happening

Measured:

- The check is a count comparison. With a layer split the runtime has one hand-off, so it expects exactly one
  stage image; every discard in this boot says `current=1` and `saved=0`.
- The engine prints both numbers itself, so it knows the image has no stage parts. It treats "0 of 1" as a
  divergence between two counts, when by its own arithmetic it is an image this runtime can never accept.
- In the on-disk format a layer-split state is *not* carried in the main session file — the writer refuses it
  (`session file: layer-split state cannot be saved`) and each stage is a separate `<stem>.stage<k>.sess`. An image
  materialized from a path that only has the main file therefore has 0 stage parts.

Hypothesis (not verified, and not the thing we fixed):

- The object reaching the validation comes from the segment tier (it is the line immediately before, in every
  occurrence we sampled: 40 of 40 in this boot, counting the 1-token chains and the 165,543-token one). Whether the
  segment chain, the archive copy, or a v1 sidecar read by a v2 runtime is the producer, we did **not** chase it.
  We fixed a *different* defect in the same neighborhood — the segment plan cut one token past the checkpoint the
  engine actually writes (cut at the `--tail-role-token` position, engine checkpoints at the `<|im_start|>` that
  opens the turn) — and the restore ladder now *sidesteps* the bad image instead of fixing its cause. That is
  declared in our notes and it is why this report exists: the sidestep hides the real cost.

## What we propose

Both are small and neither is a promise of a patch from us:

1. **Distinguish "no parts" from "a different count".** `saved=0` is a format mismatch, not a divergence; a single
   line saying so — and refusing to re-offer that file — would turn 40 silent discards into one diagnosable one.
2. **Record the split in the saved state.** The moment an image is written, its layer range is part of its identity.
   If the writer cannot represent it, say that at write time instead of at every restore.

We can send on request: the engine log for the boot above, the full config, and the 40 occurrences with their
timestamps. We are not claiming a root cause; we are reporting a whole conversation being dropped and priced at
235 s of prefill.

## What we ruled out (checked before opening this)

Searched the tracker (open and closed) for `stage count`, `"saved=0"`, `stage images`, `layer range`,
`discard snapshot`, `unreadable chain`, `stages restore`. The nearest items and why they are not this:

- **#1347** (closed/completed) `serve: a full conversation cache drops the snapshot at the physical-RAM gate
  instead of evicting to pass it (270k tokens re-read, 96 s, 32 times)` — that is the *admission* gate; this is the
  stage-count validation of an image that already exists on disk.
- **#747** (closed/completed) `Feature request: multi-conversation parking on multi-GPU layer-split` — where the
  stage count in the sidecar comes from; it does not cover an image that arrives with zero stage parts.
- **#1620** (closed/not planned) and **#2264** (open) — a cancelled request losing its entry, and the re-park every
  turn: other corners of the same cache, different failure.
- **#1572** (closed/completed) — reclaiming host caches when a SAVE needs RAM.
- Our own **#2507** (expert budget) and **#2508** (KV compaction) do not touch this path.

## What we did not test

- Windows only; we cannot compare another OS or another pair of cards.
- We did not try to force a 0-stage image (we never wrote one on purpose), so we cannot show a reproducer, only 40
  occurrences in production traffic.
- `--conversation-cache-park ram` and any single-slot configuration with the segment tier were not exercised.
- The token counts come from a live log; the readings move (see the list above).