#2589 serve: opt-in STRATA_SLOT_EXPIRE_S keeps a long conversation's batch slot from a short side request (next turn 12.7 -> 0.4 s)

本文

## Title
Issue: Related: #1906 and #1930 (agent sessions reading whole prompts again), #1920 (the same pattern one tier down: in
the checkpoint pool, short tool-call turns between two long conversations push both out, and each re-reads ~170K
tokens), #1938 (keeping more conversations' prefixes in the conversation cache). Next to #2527 / #2528 (andyhrc): they
bring a length- and age-aware placement policy to pipelined groups (`STRATA_SLOT_CACHE_AFFINITY`); this PR is the small
counterpart for the path they do not change, `--batch` without pipelined groups (one GPU, or `--batch-groups 1`).

## Summary
I run coding agents on Strata with `--batch`. Next to their long conversations, clients send short side requests (a
title, a summary, a one-line check). When every free slot holds a conversation, `pick_slot` takes the least recently
used one, so a side request can take the slot of a 40K or 160K conversation that is still going, and that
conversation's next turn reads its whole history again.

New, opt-in: `STRATA_SLOT_EXPIRE_S` (seconds, in the server's environment; off unless set). Set, a new conversation
that finds no empty slot takes an expired one first (idle that long or longer: its client has likely moved on), the
least recently used among them; else the one holding the shortest history. Unset or 0, nothing changes: the least
recently used one goes, as before (a test checks that it picks main's slot on 3,000 random slot tables). A request
still goes to the slot holding its own prefix first, and the pipelined groups keep #793's order (they return before
this rule).

<picture>
  <source media="(prefers-color-scheme: dark)" srcset="https://github.com/user-attachments/assets/bd84b68b-ce4e-4b9d-91cf-17b3e2418507">
  <img alt="Which batch slot a new conversation takes: its own history; else an empty slot; else (new) the longest idle of the slots idle STRATA_SLOT_EXPIRE_S or more; else (new) the one holding the shortest history" src="https://github.com/user-attachments/assets/594e7eb8-8637-40b0-aa25-27ba6d0fc063" width="740">
</picture>

## What changed
- `serve/server.py`: `pick_slot`'s last `min()` key (`keep_value`), and `slot_expire_s()`, which reads the variable
  once when the server starts (like `STRATA_START_RETRIES`; the config's `"env"` reaches only the engine).
- `serve/test_slot_alloc.py`: one test per rule, the measured case below, random slot tables checked against the rule
  and (off) against the old order, and the variable's values.
- `docs/BATCHING.md`: the sentence on which slot a new conversation takes, and the variable.

## Results
One RTX 5090 (Windows 11, Swift 1.5 IQ3_S, engine built from main `296b59d2`, `--batch 3`, no conversation cache), a
fresh engine per run, order main, this, main, this. A third stream decodes all along, so every request goes through a
slot. Conversation A (a 40,248-token prompt) and B (1,777 tokens) finish and stay in their slots; then a one-line side
request arrives, then A's and B's next turns:

| | main | this PR, `STRATA_SLOT_EXPIRE_S=600` |
| --- | ---: | ---: |
| slot the side request took | A's (40,251 tokens held) | B's (1,777 held) |
| A's next turn: first token | 12.72 / 12.70 s | 0.41 / 0.34 s |
| A's next turn: tokens reused | 0 of 40,273 | 40,251 of 40,273 |
| B's next turn: first token | 1.18 / 1.18 s | 1.17 / 1.18 s |

On main, A's next turn then took B's slot too (the least recently used by then), so B was read again in both arms.

Over a day of real traffic: a 21-hour timeline of my own agent traffic (9,771 requests, 4,503 conversations; ids,
times and token counts only) replayed through the slots. Prompt tokens read, and the change against the least
recently used:

| | 4 slots | 6 slots | 8 slots |
| --- | ---: | ---: | ---: |
| least recently used (main, and this PR when off) | 57.4M | 51.2M | 48.2M |
| `STRATA_SLOT_EXPIRE_S=600` | -17.8% | -20.6% | -21.1% |
| `STRATA_SLOT_EXPIRE_S=300` | -17.7% | -21.7% | -21.3% |
| shortest first, never expiring | -16.5% | -16.5% | -15.2% |
| best possible, knowing the whole timeline (a yardstick) | -18.5% | -22.6% | -22.4% |

Smaller setups: the same timeline thinned to a share of its conversations (5 random draws each), the change with
`STRATA_SLOT_EXPIRE_S=600` against the least recently used, averaged over the draws:

| conversations kept | 2 slots | 3 slots | 4 slots |
| ---: | ---: | ---: | ---: |
| 5% | -4.3% | -1.7% | -0.3% |
| 10% | -9.9% | -5.3% | -1.0% |
| 25% | -8.4% | -15.2% | -13.2% |
| 50% | -1.2% | -12.1% | -22.5% |

One slot has nothing to choose (no change). No average reads more; the worst single draw read 1.8% more (5% kept,
3 slots).

The tables send every request through a slot. Replayed again the way serve drives `--batch` (a request alone runs on
the solo path and takes a slot only when another arrives; 86-91% of these requests overlapped another), 600 s reads
17.8 / 21.2 / 21.1% less at 4 / 6 / 8 slots. With the RAM conversation cache on (`--conversation-cache-mib`), which
brings an evicted conversation back from RAM, 6 slots read 4.6% less with 4 parked conversations of 4,096 tokens or
more, 4.8% with 8 and 7.2% with 32.

Tests: `serve/test_slot_alloc` 22 of 22 (8 fail on main, and each of 11 deliberately broken versions of the rule fails
at least one); the whole serve suite: 802 OK on this branch, 785 on main (12 skipped on both).

## Extra Notes
- Python only: nothing to build for CUDA, HIP or SYCL.
- Next to #2527 / #2528: their `pick_slot` asks `slot_policy.place()` only `if gs and self.slot_cache_affinity`, i.e.
  pipelined groups with `STRATA_SLOT_CACHE_AFFINITY=1`; without pipelined groups it keeps the least recently used.
  This PR changes only that other branch, so the two do not overlap in behaviour. Both edit `pick_slot`, so whichever
  lands second needs a small rebase; with both, a one-GPU setup sets one variable, `STRATA_SLOT_EXPIRE_S`.
- Opt-in because it is measured on one GPU and one user's traffic, and it changes whose conversation loses its slot. If
  it holds up for others, a later PR can turn it on (600) by default.
- Where it loses to the old order (the expiry bounds each; a shorter one bounds them tighter):
  - a long conversation whose client has gone, or that the client restarted or compacted into a new one, keeps its
    slot until it expires while busy short chats share the other slots;
  - a long agent that waits several minutes for its human costs the busy short chats around it;
  - a client whose prompts never match what its slot holds (a date or id that changes near the top) leaves long
    histories that are never reused but stay until they expire.
- How others choose:
  - vLLM's automatic prefix caching evicts least recently used KV blocks, a request's last blocks first, and has no
    option to change it.
  - SGLang's radix cache evicts leaves, least recently used by default; `--radix-eviction-policy` also offers lfu,
    slru, priority and tlru.
  - llama.cpp's server, the closest to `--batch` slots, takes the slot with the best prompt overlap (as `pick_slot`
    does), else the least recently used one.
  - Why LRU is enough there: those caches evict blocks or tokens, so a long conversation loses only its newest blocks
    and its next turn still reuses the rest, and a prefix many requests share stays recently used. A Strata slot is
    all or nothing (the recurrent state resumes only at an exact checkpoint), so losing it costs the whole history.
- Cost-aware eviction is studied for exactly this case:
  - Marconi (MLSys 2025, arXiv 2411.19379, prefix caching for hybrid models whose SSM state is all or nothing)
    evicts by a score = recency + alpha x FLOP efficiency (the compute an entry saves per byte it holds), with alpha
    tuned on recent requests.
  - Pensieve (EuroSys 2025, arXiv 2312.05516) scores cached context by its recompute cost over the time since its
    last use.
  - GreedyDual-Size (Cao and Irani, 1997) has the best worst-case bound when misses cost different amounts.
  - This PR is the simple version: the expiry is the recency term, the history length the cost.
- I also tried GreedyDual-Size and a held-tokens-per-second rule (Pensieve-like) on the timeline: within a point or two
  of this, worse when a new agent grows next to old long conversations, and more state. Not in this PR.
- Many more slots: with the timeline laid over itself 4, 8 and 16 times (renamed, shifted copies) on 32, 64 and 128
  slots, this reads 6.1 / 4.7 / 5.0% less than the least recently used and GreedyDual-Size 7.1 / 6.8 / 7.3% less:
  within the expiry this ignores how recently a slot was used. GreedyDual-Size with this expiry on top stayed within a
  point of the better of the two at every size I tried (4 to 128 slots). A follow-up if setups with that many slots
  appear.
- Not measured on GPUs: a split with `--batch-groups 1`, and the RAM conversation cache on (replayed above: the gain
  is smaller). Pipelined groups keep #793's order here; #2527 / #2528 cover them.