#6142 [Engine] Gateway scheduling on top of the L2 session cache: cut prefill time ~90% on returning conversations — bytes-only budgets, cost-gated restore, fail-closed admission

open · @giphoo · 0 commentaires · Sur GitHub

BenchmarksServer & APIAMD / HIPNVIDIA / CUDASecurityWindowsLinux

Description

## Problem and result

**Problem.** Returning agent conversations pay the full cold-prefill cost every time the RAM cache drops them, and the engine's built-in conversation-cache admission decides silently: on a 2×RTX 5090 / 128 GiB host (#6081), 114 of 130 park attempts were refused at a physical-RAM floor while the byte budget was never reached, and none of it showed up in `/metrics` — a healthy cache and a starving one look identical, and the person tuning `--conversation-cache-mib` is tuning a knob that is not the limit.

**Result.** This gateway (stacked on #5500's engine primitives, draft until that lands) moves the policy out of the engine: budgets refuse on bytes only, restores are cost-gated per switch, admission is fail-closed with every refusal logged as an event. In three production weeks on the 48 GB AMD host: 1,971 cost-gated restores vs 4,937 switches left to engine-native reuse, restore ok-rate 94.1%, post-restore reuse p50 0.993, warm (>95% reused) true-TTFT p50 1.29 s, and the disk byte budget — not the RAM floor — is the binding limit (99.8% in use, GC-managed).

Stacks on #5500 (draft until that lands; the diff collapses to the four gateway commits once it does). This is the policy layer described in the discussion there: the engine side owns the mechanics, this gateway owns the decisions.

```mermaid
flowchart TB
    REQ["POST /v1/chat/completions"]:::io

    subgraph gw["Session Gateway — policy layer: budgets / cost gate / admission"]
        direction TB
        IDENT["① identity: family=hash(tools+system) · chain=message-hash chain"]
        QUEUE["② zero-probe cost queue: SJF+aging · single admission gate"]
        ISLIVE{"③ same-family append?"}
        BOOT["④ bootstrap save of outgoing dirty branch"]
        PICK["⑤ resolve: exact chain → deepest common-prefix ancestor → cross-family lineage → fallback"]
        PROBE["⑥ probe L0/L1 reusable prefix"]
        GATE{"⑦ cost gate<br/>C &lt; 0.8×min(A,B,D) ∧ reuse ≥ 0.5n"}
        ADMIT["⑧ fail-closed memory admission<br/>cgroup &lt; 92% max ∧ MemAvail &gt; 1.5×est ∧ PSI<br/>bounded wait → explicit degrade"]
        IDLE["⑩ idle incremental save: idle ≥ 10s ∧ ≥ 20s since last"]
    end

    subgraph engine["Engine (serve) — mechanism layer"]
        direction TB
        L0["L0 live KV / checkpoints"]
        L1["L1 PARKED RAM pool: 28 GiB + 4 GiB save reserve<br/>refuses on bytes only (slots not a constraint)"]
        REUSE["⑨ engine reuse pick: live → LCP borrow → checkpoint → PARKED best"]
        GEN["prefill the gap + decode → response"]
    end

    subgraph storage["Storage & governance"]
        direction TB
        MINI["L2-1 mini state (tmpfs)"]
        DISK["L2-2 disk archives: V3 incremental chain base+delta+ckpt<br/>global byte budget · unique-inode accounting · aliases never double-count"]
        WP["WriterPool async writes (2 GiB queue budget)<br/>index rolls forward only after generational commit"]
        EVT["event ledger: refusals / yields / evictions / reclaims → JSONL"]
    end

    REQ --> IDENT --> QUEUE --> ISLIVE
    ISLIVE -- yes --> REUSE
    ISLIVE -- no --> BOOT --> PICK --> PROBE --> GATE
    GATE -- "not worth it" --> REUSE
    GATE -- "worth it" --> ADMIT
    ADMIT -- pass --> RPP["restorep / preload as PARKED"]
    ADMIT -- "refuse (logged)" --> REUSE
    RPP --> REUSE
    L0 --> REUSE
    L1 --> REUSE
    REUSE --> GEN
    GEN -- "done · release gate" --> IDLE
    IDLE --> WP --> DISK
    DISK -.->|"restore read"| RPP
    MINI -.->|"hot_ensure"| ADMIT
    DISK -.-> EVT
    ADMIT -.-> EVT
    WP -.-> EVT
    classDef io fill:#e8f0fe;
```

## What this adds on top of #5500's primitives

- **Bytes-only budgets at every tier.** The disk tier has a global byte budget charged on unique physical objects (st_dev, st_ino), so aliases and hardlinks never double-count. The RAM pool refuses on bytes only — slot counts are deliberately not a constraint (a slot-limited pool refuses with the byte budget mostly empty; we hit exactly that before raising slots).
- **Cost-gated restore.** Every conversation switch computes A (prefill the gap from live), B (borrow from parked), C (restore from disk), D (full prefill). A restore is issued only when C &lt; (1 − margin) × min(A, B, D) and the expected reusable prefix clears a floor. Over three production weeks: 10,258 live-continuations, 4,937 switches left to engine-native reuse, 1,971 cost-gated restores.
- **Fail-closed memory admission with every refusal logged.** Before any restore: cgroup headroom, host MemAvailable, PSI sampled; bounded wait, then an explicit degrade — never a silent high-risk restore. Idle saves and readahead yield under pressure. Refusals, yields, evictions and reclaims are JSONL events — the counting layer #6081 asks for, and how we sized every budget here.
- **Managed-role mode.** `STRATA_GW_ROLE=1` registers the gateway into the existing Control process group (`strata_control/procs.py`), so the standard leftover sweep reclaims it after a Control crash exactly like a stray engine. Verified against the real record store; best-effort, never fatal, off by default.

## Configuration surface

**Turning it completely off is structural**: the gateway is a separate process in front of `--serve` — not running it, with clients pointed at `/v1` directly, leaves the engine exactly as it is without this PR (no process, no port, no engine code path touched). It builds on #5500's engine primitives, which are themselves opt-in (`STRATA_SESSION_V2`/`V3`, default off). With the gateway running, every optional behaviour has its own switch below, and the scheduler can be dropped to plain FIFO passthrough (`STRATA_GW_SCHED=off`).

Everything is env-driven with neutral defaults: relative `./var` roots, loopback endpoints, optional behaviours off. The knobs a first deployment actually touches:

**Deployment**

| env | default | meaning |
|---|---|---|
| `STRATA_GW_ENGINE_HOST` / `_PORT` | 127.0.0.1 / 8080 | where `--serve` listens |
| `STRATA_GW_BIND` / `_PORT` | 127.0.0.1 / 8191 | what this gateway listens on |
| `STRATA_GW_VAR_DIR` | ./var | storage root; snapshots / hot / logs / engine-slots live under it (each overridable) |
| `STRATA_GW_ROLE` | unset | `1` = register into the Control process group (leftover sweep reclaims it) |

**Behaviour switches (all off / standard by default)**

| env | default | meaning |
|---|---|---|
| `STRATA_GW_SCHED` | sjf | `off` = FIFO passthrough, no SJF/live reordering (L2 orchestration unchanged) |
| `STRATA_GW_TOOLJSON_UNESCAPE` | 0 | rewrite \\uXXXX escapes in tool_calls args to raw UTF-8 (a proxy should not mutate bodies unless asked) |
| `STRATA_GW_STARTUP_MANIFEST` | 0 | append a source/argv fingerprint record per boot |

**Cost model (the A/B/C/D factors; defaults calibrated on our production host)**

| env | default | meaning |
|---|---|---|
| `STRATA_COST_PREFILL_MS_PER_TOK` | 0.46 | cold prefill ≈ 2.2-2.6K tok/s |
| `STRATA_COST_RAM_MS_PER_TOK` | 0.009 | parked-pool borrow read rate |
| `STRATA_COST_L2_MS_PER_TOK` | 0.020 | disk-archive read rate |
| `STRATA_COST_RAM_FIXED_MS` / `_L2_` | 50 / 300 | fixed offsets per borrow / restore |
| `STRATA_COST_SAVE_MARGIN` | 0.20 | restore fires only when it saves ≥ 20% |
| `STRATA_COST_MIN_REUSE_FRAC` | 0.5 | expected-reuse floor |

Roughly 40 more tuning knobs (scheduler fairness, archive lifecycle, small-borrow policy, memory-admission floors like `STRATA_MEM_MIN_HOST_AVAILABLE_MIB`) ship with production defaults; the constants block at the top of `l2/gateway2.py` documents each one inline.

## Validation

Production-grade shadow run on the live host (Linux/HIP, alongside the production gateway, zero production side effects): boot, endpoint resolution, directory bootstrap, `/__gw/status`, non-stream and SSE requests, tool_calls UTF-8 passthrough, cost-model arithmetic bit-identical to the production implementation, idle saveinc archiving two sessions to an isolated root, role registration/unregistration against `procs.py`. One first-boot nit found and fixed (absent engine slot dir now starts with an empty index instead of an Errno line). Historical engine-side evidence (359-check suite, 51-request rotation) is inherited from #5500.

## Limits

- Validated on Linux/HIP only; the cgroup/PSI admission checks are Linux-specific (absent on Windows, admission degrades to MemAvailable-only where available).
- The scheduler (SJF + live-continuation preference + aging) can be switched to FIFO passthrough (`STRATA_GW_SCHED=off`) but not removed.
- No auth/CORS: it sits beside `--serve` on loopback by default, same trust domain as the engine itself.

Retarget to a clean base once #5500 lands.

Sur le site

Liens install, modèles, releases.