#2528 Slot placement policy for STRATA_SLOT_CACHE_AFFINITY (2/2, depends on #2527): protection by history length, retention scores, value decay, no wait

Descrição

## Title
Issue: none open that I found; part 2 of #2527.


## How this PR relates to #2527 (please read first)

This is **part 2 of 2** and **depends on #2527**.

- **#2527 (part 1)**: the engine changes and the server side, with `serve/slot_policy.py` as an interface only.
- **This PR**: our implementation of that interface, including expired-history parking and quadratic idle-value decay.
  Small supporting server changes preserve the last actual use across parking and skip low-value main histories.
  The branch includes the mechanism commits; review the policy changes after #2527.
- **#2589** (SuperDistance) is the small counterpart for the path this PR does not change: `--batch` without
  pipelined groups (one GPU, or `--batch-groups 1`).
- **Merge order**: #2527 first, then this one. The policy is also applied to our production checkout;
  the time-decay change awaits the next server restart.
- **If you prefer other rules than ours**: keep #2527 and replace `serve/slot_policy.py`; the thresholds below are
  parameters, so many variations need no code at all.

## Summary

The policy behind `serve/slot_policy.py`'s interface: which idle slot a request runs in, which history to give up,
how long one is protected, where the solo path's conversation is parked. The policy is kept identical in this PR and our production checkout on the machine described in #2527.
This revision has run in production since 2026-10-15 (GPU checks below).

Plain rules cannot tell which history to give up when a new conversation needs a slot and none is free of value: the
one used longest ago may be a 90K-token conversation whose user is reading the answer, the newest a side request's 300
tokens. This policy weighs length and age, and every threshold is a parameter in the server config's `"env"`, so most
other behaviour is a matter of setting them:

| Variable | Default | What it sets |
| --- | --- | --- |
| `STRATA_SLOT_TTL_MAX_S` | 120 | protection at 262144 tokens; the whole curve scales with it |
| `STRATA_SLOT_EXPIRED_KEEP_SCORE` | 0.01 | an expired history whose current decayed value is at least this goes last |
| `STRATA_SLOT_IDLE_VALUE_ZERO_S` | 1800 | total idle age, including protection, when value reaches zero; 0 disables decay |
| `STRATA_SLOT_PARK_VALUE_RATIO` | 2 | a park takes an expired history worth keeping only when the main session's is worth this many times more; 0: never |
| `STRATA_SLOT_PARK_MIN_VALUE` | 0.01 | a solo conversation whose value at its idle age is below this is not parked |
| `STRATA_SLOT_PARK_KEPT_SHARE` | 0.98 | a slot holding this share of the solo conversation already keeps it (no park) |

## The rules

- **Protection by length**: 0.1 s at 500 tokens, 2 s at 2K, 10 s at 8K, 45 s at 32K, 65 s at 50K, 90 s at 100K, 120 s
  at 262144 (piecewise linear).
- **Retention** = (n / 50000)^0.7 x (n / (n + 2000))^2 x (0.25 + 0.75 x freshness), freshness = 1 - age / TTL, 0 once
  expired; overwrite score = 1 / (1 + retention). A fresh 50K history is worth about 1, a 500-token one 0.002.
- **Value after expiry** = (n / 50000)^0.7 x (n / (n + 2000))^2 x 0.25 x (1 - x^2),
  where x = clamp((age - TTL) / (idle_value_zero_s - TTL), 0, 1). Age is measured from the end of the last actual
  use, including the protection period. Loss per second increases throughout the decay window. With the default
  1800 s total idle limit, value is zero at 30 minutes for every supported length. Protection takes precedence
  if a configured limit is shorter than the TTL; 0 disables decay. Allocation and parking use the same value.
  This replaces the earlier exponential half-life parameter. The curve is the requested retention rule;
  no new GPU workload measurement establishes an optimal lifetime.
- **Main-session parking** first checks this value against `STRATA_SLOT_EXPIRED_KEEP_SCORE`; a zero or below-threshold
  main history is not parked even if a blank slot exists. A park preserves the main history's last actual use,
  so copying cannot renew its protection or idle lifetime. Continued use refreshes the timestamp on completion.
  A zero-value history is still eligible for prefix reuse until it is overwritten; the score controls replacement
  and parking, never cache validity or matching. No timer deletes history at 30 minutes.
- **Blank slots** (empty, invalid, or a superseded copy) count as expired and worthless; **an expired history is not
  deleted** - it still matches until it is overwritten.
- **A continued conversation** runs in its slot or in a blank slot of a less loaded group, **never in a slot that
  holds another conversation**; near matches (2% of the best, one second of the prompt rate, 4096 tokens) are sources.
  An earlier version that put group balance first moved two idle Claude Code conversations into each other's slot on
  every turn, and each turn read 60-100K tokens from token 0.
- **A new conversation** takes what is not worth keeping first (blank, or an expired history below
  `STRATA_SLOT_EXPIRED_KEEP_SCORE`), then by group load, then the lowest value (a blank slot first at an equal
  value). A park uses the same order (`reclaim_slot`).
- **No wait**: every history protected - overwrite the least loss at once (losses within 10% by group load). An
  earlier version waited up to 5 s first, once per request; with more requests than slots each wait ran its full 5 s
  with a slot idle and ended in the same overwrite (7 of 7 in a 15-minute test with 8 clients on 6 slots).
- **Park**: not when its value at its idle age is below `STRATA_SLOT_PARK_MIN_VALUE` (no separate length cut-off; a
  copy of 10K-170K tokens took 0.17-2.56 s in production, every slot paused), not when a slot holds a valid copy of
  `STRATA_SLOT_PARK_KEPT_SHARE` of it already (the copy breaks even with re-reading a 1-4% tail at 1,000-3,000 tok/s);
  the conversation's own older copy; else the new-conversation order above, where a history worth keeping counts only
  when the main session's conversation is worth `STRATA_SLOT_PARK_VALUE_RATIO` times as much. Never a protected slot;
  nowhere suitable: no park.
- On an engine without `slot_padding_safe=1`, starting a row above a protected idle history costs its retention too.

## Results

The first table and the park line predate this revision. No throughput or cache-hit measurement over a working day
is claimed for the 30-minute default; the scenario checks of this revision follow them.

3 x RTX A4000 16 GB, Qwen3.8-Flash-Next IQ3_S, 0.1.44.1, `--batch 6` in 3 groups, `STRATA_PIPELINE_ACTIVE_ROWS=1`.
Conversation A (37K tokens) parked, two clients generating 1,500-token answers in a loop (batch mode stays on),
conversations B (9K) and C (7.9K) taking turns about every 30 s - past their protection of about 11 and 10 s. Tokens
reused on turns 1-3:

| Ordering of expired histories | B | C |
| --- | --- | --- |
| by group load first (an earlier version) | 0 every turn | 0 every turn |
| + one still worth keeping goes last (this PR) | all but the ~20 new | 0 every turn |
| + wait for a cheaper slot about to leave its protection (removed) | all but the ~20 new | all but the ~20 new |

In the first row a new turn took the other conversation's expired slot for group balance, so B and C overwrote each
other; in the second, C went to a side request that arrived a second after another one finished, while that one's
1.5K-token history was about to leave its protection. The third row waited for that protection to end, under the same
once-per-request wait; it went with that wait. This PR accepts the case: a protected history is not overwritten while
an expired one can be.

Park: a 37,359-token conversation's next turn after a side request reused 37,352 tokens and took about 1 s (0 and
14 s without), with the same greedy output.

This revision, same machine and settings, `STRATA_SLOT_IDLE_VALUE_ZERO_S=150` so that decay shows within minutes:

| Check | Result |
| --- | --- |
| a park keeps the last use | A (11.9K) parked 27.0 s after its turn ended: the slot's age is 27.0 s, not 0 |
| a main history past the zero age | idle 160 s: the next solo request records `park_skip` (main_low_value), no slot taken |
| every slot worth keeping, main worth less than twice as much | six slots of 10,062 tokens (value 0.056), main 12.9K: no park |
| main worth more than twice as much | main 42.1K (0.202): parked into the slot with the lowest value (the oldest); its next turn reused 42,092 of 42,119 tokens, 0.7 s |

## How it was checked

`serve/test_slot_alloc.py` (the policy's own cases, including the production ones above), `serve/test_parallel.py`
(the trace fields),
`serve/test_cache_trace.py`. New cases cover accelerating loss, expiry continuity and the total idle cutoff,
threshold crossing for 8K through 262K, refreshing age on use, zero-value prefix matching, main-session timestamp
recording, pre-park threshold checks, age inheritance during parking, busy/excluded targets and custom/disabled/
invalid settings. Protected histories retain their protection if a configured total idle limit is shorter.

`python -m unittest serve.test_slot_alloc serve.test_cache_trace serve.test_parallel -q`: 104 tests pass (56.1 s).
`place()` before and after the shared reclaim order chose the same slot in 200,000 random slot tables.
Historical full-suite check before the decay change: `python -m unittest discover -s serve -t .` passes except
`serve.test_responses.NoLeakedSampler`, which fails on upstream main on this machine too.

## Extra Notes

Only the thresholds have names; if you want a different trade-off (for example a fixed protection time, or plain
least-recently-used), `place()` is the one function to change, and the server side does not need to.

🤖 Generated with [Claude Code](https://claude.com/claude-code)