#6142 [Engine] Gateway scheduling on top of the L2 session cache: cut prefill time ~90% on returning conversations — bytes-only budgets, cost-gated restore, fail-closed admission
open · @giphoo · 0 comentarios · En GitHub
BenchmarksServer & APIAMD / HIPNVIDIA / CUDASecurityWindowsLinux
Descripción
## Problem and result
**Problem.** Returning agent conversations pay the full cold-prefill cost every time the RAM cache drops them, and the engine's built-in conversation-cache admission decides silently: on a 2×RTX 5090 / 128 GiB host (#6081), 114 of 130 park attempts were refused at a physical-RAM floor while the byte budget was never reached, and none of it showed up in `/metrics` — a healthy cache and a starving one look identical, and the person tuning `--conversation-cache-mib` is tuning a knob that is not the limit.
**Result.** This gateway (stacked on #5500's engine primitives, draft until that lands) moves the policy out of the engine: budgets refuse on bytes only, restores are cost-gated per switch, admission is fail-closed with every refusal logged as an event. In three production weeks on the 48 GB AMD host: 1,971 cost-gated restores vs 4,937 switches left to engine-native reuse, restore ok-rate 94.1%, post-restore reuse p50 0.993, warm (>95% reused) true-TTFT p50 1.29 s, and the disk byte budget — not the RAM floor — is the binding limit (99.8% in use, GC-managed).
Stacks on #5500 (draft until that lands; the diff collapses to the four gateway commits once it does). This is the policy layer described in the discussion there: the engine side owns the mechanics, this gateway owns the decisions.
```mermaid
flowchart TB
REQ["POST /v1/chat/completions"]:::io
subgraph gw["Session Gateway — policy layer: budgets / cost gate / admission"]
direction TB
IDENT["① identity: family=hash(tools+system) · chain=message-hash chain"]
QUEUE["② zero-probe cost queue: SJF+aging · single admission gate"]
ISLIVE{"③ same-family append?"}
BOOT["④ bootstrap save of outgoing dirty branch"]
PICK["⑤ resolve: exact chain → deepest common-prefix ancestor → cross-family lineage → fallback"]
PROBE["⑥ probe L0/L1 reusable prefix"]
GATE{"⑦ cost gate<br/>C < 0.8×min(A,B,D) ∧ reuse ≥ 0.5n"}
ADMIT["⑧ fail-closed memory admission<br/>cgroup < 92% max ∧ MemAvail > 1.5×est ∧ PSI<br/>bounded wait → explicit degrade"]
IDLE["⑩ idle incremental save: idle ≥ 10s ∧ ≥ 20s since last"]
end
subgraph engine["Engine (serve) — mechanism layer"]
direction TB
L0["L0 live KV / checkpoints"]
L1["L1 PARKED RAM pool: 28 GiB + 4 GiB save reserve<br/>refuses on bytes only (slots not a constraint)"]
REUSE["⑨ engine reuse pick: live → LCP borrow → checkpoint → PARKED best"]
GEN["prefill the gap + decode → response"]
end
subgraph storage["Storage & governance"]
direction TB
MINI["L2-1 mini state (tmpfs)"]
DISK["L2-2 disk archives: V3 incremental chain base+delta+ckpt<br/>global byte budget · unique-inode accounting · aliases never double-count"]
WP["WriterPool async writes (2 GiB queue budget)<br/>index rolls forward only after generational commit"]
EVT["event ledger: refusals / yields / evictions / reclaims → JSONL"]
end
REQ --> IDENT --> QUEUE --> ISLIVE
ISLIVE -- yes --> REUSE
ISLIVE -- no --> BOOT --> PICK --> PROBE --> GATE
GATE -- "not worth it" --> REUSE
GATE -- "worth it" --> ADMIT
ADMIT -- pass --> RPP["restorep / preload as PARKED"]
ADMIT -- "refuse (logged)" --> REUSE
RPP --> REUSE
L0 --> REUSE
L1 --> REUSE
REUSE --> GEN
GEN -- "done · release gate" --> IDLE
IDLE --> WP --> DISK
DISK -.->|"restore read"| RPP
MINI -.->|"hot_ensure"| ADMIT
DISK -.-> EVT
ADMIT -.-> EVT
WP -.-> EVT
classDef io fill:#e8f0fe;
```
## What this adds on top of #5500's primitives
- **Bytes-only budgets at every tier.** The disk tier has a global byte budget charged on unique physical objects (st_dev, st_ino), so aliases and hardlinks never double-count. The RAM pool refuses on bytes only — slot counts are deliberately not a constraint (a slot-limited pool refuses with the byte budget mostly empty; we hit exactly that before raising slots).
- **Cost-gated restore.** Every conversation switch computes A (prefill the gap from live), B (borrow from parked), C (restore from disk), D (full prefill). A restore is issued only when C < (1 − margin) × min(A, B, D) and the expected reusable prefix clears a floor. Over three production weeks: 10,258 live-continuations, 4,937 switches left to engine-native reuse, 1,971 cost-gated restores.
- **Fail-closed memory admission with every refusal logged.** Before any restore: cgroup headroom, host MemAvailable, PSI sampled; bounded wait, then an explicit degrade — never a silent high-risk restore. Idle saves and readahead yield under pressure. Refusals, yields, evictions and reclaims are JSONL events — the counting layer #6081 asks for, and how we sized every budget here.
- **Managed-role mode.** `STRATA_GW_ROLE=1` registers the gateway into the existing Control process group (`strata_control/procs.py`), so the standard leftover sweep reclaims it after a Control crash exactly like a stray engine. Verified against the real record store; best-effort, never fatal, off by default.
## Configuration surface
**Turning it completely off is structural**: the gateway is a separate process in front of `--serve` — not running it, with clients pointed at `/v1` directly, leaves the engine exactly as it is without this PR (no process, no port, no engine code path touched). It builds on #5500's engine primitives, which are themselves opt-in (`STRATA_SESSION_V2`/`V3`, default off). With the gateway running, every optional behaviour has its own switch below, and the scheduler can be dropped to plain FIFO passthrough (`STRATA_GW_SCHED=off`).
Everything is env-driven with neutral defaults: relative `./var` roots, loopback endpoints, optional behaviours off. The knobs a first deployment actually touches:
**Deployment**
| env | default | meaning |
|---|---|---|
| `STRATA_GW_ENGINE_HOST` / `_PORT` | 127.0.0.1 / 8080 | where `--serve` listens |
| `STRATA_GW_BIND` / `_PORT` | 127.0.0.1 / 8191 | what this gateway listens on |
| `STRATA_GW_VAR_DIR` | ./var | storage root; snapshots / hot / logs / engine-slots live under it (each overridable) |
| `STRATA_GW_ROLE` | unset | `1` = register into the Control process group (leftover sweep reclaims it) |
**Behaviour switches (all off / standard by default)**
| env | default | meaning |
|---|---|---|
| `STRATA_GW_SCHED` | sjf | `off` = FIFO passthrough, no SJF/live reordering (L2 orchestration unchanged) |
| `STRATA_GW_TOOLJSON_UNESCAPE` | 0 | rewrite \\uXXXX escapes in tool_calls args to raw UTF-8 (a proxy should not mutate bodies unless asked) |
| `STRATA_GW_STARTUP_MANIFEST` | 0 | append a source/argv fingerprint record per boot |
**Cost model (the A/B/C/D factors; defaults calibrated on our production host)**
| env | default | meaning |
|---|---|---|
| `STRATA_COST_PREFILL_MS_PER_TOK` | 0.46 | cold prefill ≈ 2.2-2.6K tok/s |
| `STRATA_COST_RAM_MS_PER_TOK` | 0.009 | parked-pool borrow read rate |
| `STRATA_COST_L2_MS_PER_TOK` | 0.020 | disk-archive read rate |
| `STRATA_COST_RAM_FIXED_MS` / `_L2_` | 50 / 300 | fixed offsets per borrow / restore |
| `STRATA_COST_SAVE_MARGIN` | 0.20 | restore fires only when it saves ≥ 20% |
| `STRATA_COST_MIN_REUSE_FRAC` | 0.5 | expected-reuse floor |
Roughly 40 more tuning knobs (scheduler fairness, archive lifecycle, small-borrow policy, memory-admission floors like `STRATA_MEM_MIN_HOST_AVAILABLE_MIB`) ship with production defaults; the constants block at the top of `l2/gateway2.py` documents each one inline.
## Validation
Production-grade shadow run on the live host (Linux/HIP, alongside the production gateway, zero production side effects): boot, endpoint resolution, directory bootstrap, `/__gw/status`, non-stream and SSE requests, tool_calls UTF-8 passthrough, cost-model arithmetic bit-identical to the production implementation, idle saveinc archiving two sessions to an isolated root, role registration/unregistration against `procs.py`. One first-boot nit found and fixed (absent engine slot dir now starts with an empty index instead of an Errno line). Historical engine-side evidence (359-check suite, 51-request rotation) is inherited from #5500.
## Limits
- Validated on Linux/HIP only; the cgroup/PSI admission checks are Linux-specific (absent on Windows, admission degrades to MemAvailable-only where available).
- The scheduler (SJF + live-continuation preference + aging) can be switched to FIFO passthrough (`STRATA_GW_SCHED=off`) but not removed.
- No auth/CORS: it sits beside `--serve` on loopback by default, same trust domain as the engine itself.
Retarget to a clean base once #5500 lands.En el sitio
Enlaces a install, modelos, releases.