#2597 [Feature]: degrade the conversation cache across levels instead of deleting the oldest (RAM -> fast M.2 -> cold disk, each level switchable)

本文

## What exists today

The conversation cache's disk tier has exactly one full policy:

- `--conversation-cache-spill-when-full` accepts two values, `evict-oldest` (the default) and `reject`.
- With `evict-oldest`, the GC **deletes** the oldest entry to make room. There is no pin, no protected set, and no
  second (colder) level to move it to.
- Start-up line from the boot we run today:

```text
strata serve: conversation cache: spill dir ready (20 conversations, 20225 MiB, when-full=evict-oldest, max-age=0 d, spill-on=park, divergence=4096 tok, park-throttle=0 s, 0 oversized kept, 58 stale kept, 0 foreign kept, 0 orphans kept, 0 disk evictions, 0 age evictions)
```

- We have seen the FIFO work: 101 disk evictions counted in a measurement window on 2026-10-14, and the log's
  cumulative `evictions=` counter reaches 132. (In the boot above the counter is 0 because the budget, 460,800 MiB,
  is far above what is in use, 20,225 MiB.)

## What we ask: a cascade that degrades instead of deleting

The entry that leaves a level **enters the next one**; only the last level discards.

| level | destination | switch | role |
|---|---|---|---|
| 1 | RAM | `--conversation-cache-mib X` (`0` = off) | hot |
| 2 | fast M.2, a few GB | `--conversation-cache-disk-mib` (`0` = off) | warm |
| 3 | cold M.2/SATA SSD, hundreds of GB | `--conversation-cache-cold-dir` + `--conversation-cache-cold-mib` (`0` = off) | cold |
| - | floor | `--conversation-cache-min-free-mib` | **stays as it is** |

Two properties we care about, and they are the whole request: **each level can be switched off independently**, and
the RAM floor is not part of the cascade (it protects the rest of the machine, it is not a parking level).

## Why the existing `archive` cannot be the cold level

The archive we already run (`--conversation-cache-archive-mode state`, `--conversation-cache-archive-keep 4`) is
history with a contract: an archived state is never a reuse candidate — it exists so the state is not lost, not to
make a reload cheap. Level 3 has to be a real tier: its own GC, its own match index, and a restore path.

```text
strata serve: conversation cache: archive ready (state, 9 archived, 7461 MiB, mib=8192, keep=4, max-age=0 d, 0 archive evictions [0 by age])
```

To be explicit about the source of that claim: it is our reading of the archive path, not a quote from your docs. If
the archive *is* meant to be a reuse source, say so and this section is moot.

## Design by media type, not by drive letter

Level 2 should be the volume the setup bench measures as the fastest with room; level 3 the next by sequential
speed, or the one with the most free space. On Windows the setup already probes disks, so an `auto` that reads
volumes (sequential read/write, free space, bus type NVMe/SATA) is the piece that assigns the levels — not a
hand-written list. Re-running it when a disk is added should be automatic.

We are not asking you to design that part blind; here is what it looks like on this machine, for scale:

- `D:` is a 1.9 TB NVMe SSD with ~597 GiB free, measured filling the expert arena at 6.03 GiB/s and restoring
  2.9 GB in 274 ms (the latter assisted by the OS file cache).
- `C:` has ~44 GB free and hosts the pagefile, so it is not a candidate for a warm level.
- The operator's `F:` is excluded by policy.

So today levels 2 and 3 would land on the same volume — the cascade degrades inside one disk until a second medium
exists. That is fine as a design, but it belongs in the docs, and it is why we would rather have two switchable
levels than one level plus an assumption.

## Why we ask (what the numbers buy today)

- The tier is not decorative here: 1,294 restores and ~90.5 M tokens restored between 2026-10-07 and 2026-10-14, the
  largest a 171,516-token context in 656.9 ms against 158.6 s cold (24x).
- `evict-oldest` deletes exactly the copy that makes the next reload cheap. The entry goes to the archive only when
  the archive is enabled, and per the contract above that copy is not reusable.
- A smaller warm budget is the realistic setting on a workstation that shares the disk with everything else, and
  under a small budget every restart silently trades history for the newest few conversations.

## What we owe you as a counter-argument

With a large disk budget (450 GiB in our config) the FIFO already keeps the same hot set plus everything else, and a
second level buys nothing. The cascade matters when the fast volume is small, when the two media differ in speed, or
when the operator wants history kept without it competing with the hot set. **We are not claiming a measured
regression from `evict-oldest`** — this is a design request with an operator-side motivation, not a bug report.

## What we do not claim

- Not implemented, not benchmarked, and we are not sending code for it.
- `--conversation-cache-cold-dir` / `--conversation-cache-cold-mib` do not exist in our build.
- The 101 evictions are a count, not a cost: we did not measure how much prefill the current eviction policy caused.
- Windows only.

## What we ruled out (checked before opening this)

- **#47** (closed/completed) `Spill kv-cache to nvme` — the origin of the disk tier.
- **#847** (closed/completed) `Allow system ram to be used instead of SSD when available.` — the RAM/disk ordering.
- **#661** (closed/completed) `Proposal: save and restore a conversation to disk` and **#1444** (closed/not planned)
  `can kvcache be saved and reused on disk?` — the save/restore path itself.
- **#1572** (closed/completed) reclaiming disposable host caches on SAVE, and **#1347** (closed/completed) evicting
  to pass the physical-RAM gate instead of dropping the snapshot — both are *eviction to pass a gate*, not
  *demotion between levels*.
- **#2033** (open) `MADV_COLD` for swap and **#1900** (open) NUMA cold pools — colder placement, not a cascade.
- **#2275** (open) and **PR #2050** — the read path that stops the machine while a cold read runs. A cascade that
  pushes reads onto a slower medium makes that path longer; same direction, different fix, and worth knowing before
  adding a level.
- Searched for `evict`, `when-full`, `demote`, `cascade`, `cold tier`, `second tier`, `spill`: none of the above
  asks for "the entry that leaves level N enters level N+1, and only the last level discards".

## What we did not test

- No implementation, so no measurement of the cascade.
- We did not measure the cost of the current eviction on reload (we have the count, not the price).
- We did not test the disk tier under a small budget for long enough to see the FIFO delete a conversation we then
  wanted back; the 101 evictions come from the monitor we run, not from a reload experiment.