#5820 [Engine] Multiple API keys with per-key concurrency limits and reserved capacity

open · @NoBirdWasHam · 0 commentaires · Sur GitHub

Setup & installServer & APIAMD / HIPNVIDIA / CUDADocumentationWindowsLinux

Description

## Problem (#5004)

Running Strata as a shared local server: with `--parallel 2`, one client (Hermes Agent) can submit enough concurrent requests to occupy both slots, so a latency-sensitive Home Assistant request waits behind work that was not its own. `api_key` can already accept several keys (#1344), but the keys are anonymous: there is no way to limit a key's concurrency independently or keep capacity for one client.

## What this adds (opt-in; without the block, nothing is counted)

A config block, keyed by the key itself:

```json
"api_keys": {
  "hermes-key":         {"name": "hermes", "max_concurrent": 1},
  "home-assistant-key": {"name": "home assistant", "max_concurrent": 1, "reserved_capacity": 1}
}
```

- Every entry is a key the server accepts (it joins the `api_key` set; policy keys are checked against `telemetry_read_key` like the others).
- `max_concurrent`: how many requests of that key the server holds at once (from entering `Service.run()` to the end: queue wait, prompt read and generation all count). More of them wait their turn.
- `reserved_capacity`: slots of a `parallel` server that only this key may sit on. Every other key's reach shrinks by it - a reserved slot waits empty rather than being borrowed. No preemption: a request already decoding keeps its slot; the reservation holds at the next admission.
- A key the block does not name (a plain `api_key`) shares `parallel - all reservations`.
- A wait at a key's limit is the queue's own kind of wait: `queue_timeout` bounds it - per wait: a request that waits at its key's limit and then in the engine queue can spend up to twice `queue_timeout` before its 429 - and the same `429 queue_wait_timeout` answers its expiry; a client that hangs up mid-wait ends it (`ClientGone`, the existing disconnect path). No new HTTP error shapes.
- `GET /status` gains `"api_keys"`: each entry's name, in flight, limits - names and counts, never the keys.
- A contradictory block stops the start with the reason (same policy as `origins_of`: better to refuse than to limit less or more than meant).

## Why this shape

- Everything lives in `terminal/serve/server.py` + `terminal/serve/server_parts/http_api.py` (post-0.2.16 layout; the key helpers live in `http_api.py`). The gate is wrapped around `Service.run()` - the one choke point every inference path passes (chat/completions, messages, responses; stream and non-stream; MCP rounds count as their request). The engine's slot scheduler (`_take_control`, `pick_slot`) is not touched, so **no C++/engine code changes and no backend is affected by this diff**.
- Identity: `matched_key()` answers *which* configured key passed - the same both-encoding, every-key-compared comparison as before (`key_matches` now delegates to it, so #725 and #213 semantics are unchanged). The matched key rides the request's cancel Event (`cancel._strata_key`, stamped at client-watch time, alongside the other per-request attributes).
- The limits follow the engine's live `batch`, so a `vram_fit`-shrunk `parallel` is respected; with `"replicas"`, `ReplicaEngine.batch` is the sum of the replicas' slots, so a per-key limit bounds the whole service, not one replica.

## Known limits (documented in `docs/DETAILS.md`)

- No preemption: `reserved_capacity` protects the next admission, never a slot a request already holds.
- A reserved slot is held for its key even while that key is idle.
- `POST /v1/images/generations` does not pass `Service.run()` and is not counted.
- A request parked at the gate has already sent its SSE headers and sends nothing until it is admitted or expires; parked requests are not in the `queued` status counts, and a key counts in `api_keys` only once admitted. The gate's own expiries are not in the `queue_timeouts` metric.
- Reservations are checked against the `parallel` the server starts with; if the engine's live slots later shrink below the reservations (VRAM fit), keys without a reservation can lose their reach until the next start.

## Validation (head `ecaf5ef` = the same single squashed commit, cherry-picked clean onto `main` at `c6f6427` (v0.2.19); Windows 11)

The head is the same 485-insertion diff as at `d32c378ba`, re-landed on current main by cherry-pick: zero conflicts onto the 0.2.19 base, insertion seams inspected on the result (the `docs/DETAILS.md` config-table row and per-key bullet still sit inside the API-key section), upstream main independently contains none of `matched_key` / `KeyLimits` / `api_key_limits_of` (no competing implementation; unchanged at 0.2.19). The 55-commit 0.2.19 window touches `server.py` again; the cherry-pick lands clean and the CI-scoped failure sets below match plain upstream exactly. Port note for the 0.2.16 layout in the thread, 18 NOV.

- Feature suite `python -m pytest terminal/serve/test_api_keys.py -q` on Python 3.14 (dev set): **19 passed** at this head
- Upstream tests for the touched areas, at this head: `test_admit_order` + `test_runconfig` — **13 passed** (the three files together: 32 passed + 41 subtests)
- CI-scoped battery, `tools/check --only py-tests,py-serve,py-tools` (CI's own steps) on this head: **py-serve 2,079 passed / 10 failed / 69 skipped (25m), py-tools 1,841 passed / 7 failed / 77 skipped, py-tests 1 failed (import)**. A **control run on plain upstream `c6f6427`** (same interpreter, same session) has **zero head-only failures in all three steps** (the control shows one extra flake in the same `test_vision_cpu_fallback` family; membership churns run-to-run; the control passes 20 fewer — this PR's 19 feature tests plus one flake that did not fire on it). None introduced by this diff; the families are the known Windows vision/subprocess ones plus a POSIX-only import in `tests.split.test_split_engine`.
- Python 3.11 minimal-serving-runtime (`requirements.txt` only): `python -m unittest serve.test_api_keys` — **19 tests, OK** (Python 3.11, `requirements.txt` only, re-run at this head)
- `tools/check --only docs` — **passed at this head** (6 checks, 0 failed, incl. the settings-register check over the config tables)
- `tests/snapshot` — the drafting round-trip snapshot tests fail identically on plain upstream on this Windows box (byte-for-byte same counts with and without this PR's commit, established on the prior head); this diff touches no launch path the snapshot suite drives differently
- **Real engine, not mock** (RTX 3090 Ti, `parallel 2`, two live clients): the startup line reports the configured keys; `/status` shows the per-key view (names, never keys); concurrent saturation streams on one capped key serialize at the gate (183 live samples on the production server, engine 0.1.62: in-flight never exceeded the key's max); a latency-sensitive client on the `reserved_capacity` key answered immediately throughout; the `api_keys` block survived repeated `setup`/update rewrites via `carry_over` (two production updates since 15 NOV)

Not run at this head: the ubuntu-24.04 leg of `py-serve` (no Linux box here) — CI covers it once the runs are approved.

## Status note

Maintainer response of 20 NOV: kept open as a product decision, not scheduled for 0.2.17; upstream remains without an implementation at 0.2.19 (`c6f6427`). Validation is kept current at every head rebase; the fork branch stays rebased on main per the running production deployment.

Sur le site

Liens install, modèles, releases.