#5820 [Engine] Multiple API keys with per-key concurrency limits and reserved capacity
open · @NoBirdWasHam · 0 评论 · 去 GitHub 看
Setup & installServer & APINVIDIA / CUDADocumentationWindowsLinux
说明
## Problem (#5004)
Running Strata as a shared local server: with `--parallel 2`, one client (Hermes Agent) can submit enough concurrent requests to occupy both slots, so a latency-sensitive Home Assistant request waits behind work that was not its own. `api_key` can already accept several keys (#1344), but the keys are anonymous: there is no way to limit a key's concurrency independently or keep capacity for one client.
## What this adds (opt-in; without the block, nothing is counted)
A config block, keyed by the key itself:
```json
"api_keys": {
"hermes-key": {"name": "hermes", "max_concurrent": 1},
"home-assistant-key": {"name": "home assistant", "max_concurrent": 1, "reserved_capacity": 1}
}
```
- Every entry is a key the server accepts (it joins the `api_key` set; policy keys are checked against `telemetry_read_key` like the others).
- `max_concurrent`: how many requests of that key the server holds at once (from entering `Service.run()` to the end: queue wait, prompt read and generation all count). More of them wait their turn.
- `reserved_capacity`: slots of a `parallel` server that only this key may sit on. Every other key's reach shrinks by it - a reserved slot waits empty rather than being borrowed. No preemption: a request already decoding keeps its slot; the reservation holds at the next admission.
- A key the block does not name (a plain `api_key`) shares `parallel - all reservations`.
- A wait at a key's limit is the queue's own kind of wait: `queue_timeout` bounds it - per wait: a request that waits at its key's limit and then in the engine queue can spend up to twice `queue_timeout` before its 429 - and the same `429 queue_wait_timeout` answers its expiry; a client that hangs up mid-wait ends it (`ClientGone`, the existing disconnect path). No new HTTP error shapes.
- `GET /status` gains `"api_keys"`: each entry's name, in flight, limits - names and counts, never the keys.
- A contradictory block stops the start with the reason (same policy as `origins_of`: better to refuse than to limit less or more than meant).
## Why this shape
- Everything lives in `terminal/serve/server.py` + `terminal/serve/server_parts/http_api.py` (post-0.2.16 layout; the key helpers live in `http_api.py`). The gate is wrapped around `Service.run()` - the one choke point every inference path passes (chat/completions, messages, responses; stream and non-stream; MCP rounds count as their request). The engine's slot scheduler (`_take_control`, `pick_slot`) is not touched, so **no C++/engine code changes and no backend is affected by this diff**.
- Identity: `matched_key()` answers *which* configured key passed - the same both-encoding, every-key-compared comparison as before (`key_matches` now delegates to it, so #725 and #213 semantics are unchanged). The matched key rides the request's cancel Event (`cancel._strata_key`, stamped at client-watch time, alongside the other per-request attributes).
- The limits follow the engine's live `batch`, so a `vram_fit`-shrunk `parallel` is respected; with `"replicas"`, `ReplicaEngine.batch` is the sum of the replicas' slots, so a per-key limit bounds the whole service, not one replica.
## Known limits (documented in `docs/DETAILS.md`)
- No preemption: `reserved_capacity` protects the next admission, never a slot a request already holds.
- A reserved slot is held for its key even while that key is idle.
- `POST /v1/images/generations` does not pass `Service.run()` and is not counted.
- A request parked at the gate has already sent its SSE headers and sends nothing until it is admitted or expires; parked requests are not in the `queued` status counts, and a key counts in `api_keys` only once admitted. The gate's own expiries are not in the `queue_timeouts` metric.
- Reservations are checked against the `parallel` the server starts with; if the engine's live slots later shrink below the reservations (VRAM fit), keys without a reservation can lose their reach until the next start.
## Validation (head `b63d61c` = the same single squashed commit, cherry-picked clean onto `main` at `6e38f58` (v0.2.18); Windows 11)
The head is the same 485-insertion diff as at `d32c378ba`, re-landed on current main by cherry-pick: zero conflicts onto the 0.2.18 base, insertion seams inspected on the result (the `docs/DETAILS.md` config-table row and per-key bullet still sit inside the API-key section), upstream main independently contains none of `matched_key` / `KeyLimits` / `api_key_limits_of` (no competing implementation; unchanged at 0.2.18). Upstream's 224 new commits touch `terminal/serve/server.py` in 30 hunks, none of which intersect this diff's four anchor areas, and leave `server_parts/http_api.py` untouched. Port note for the 0.2.16 layout in the thread, 18 NOV.
- Feature suite `python -m pytest terminal/serve/test_api_keys.py -q` on Python 3.14 (dev set): **19 passed** at this head
- Upstream tests for the touched areas, at this head: `test_admit_order` + `test_runconfig` — **13 passed** (the three files together: 32 passed + 41 subtests)
- CI-scoped battery, `tools/check --only py-tests,py-serve,py-tools` (CI's own steps) on this head: **py-serve 2,061 passed / 11 failed / 68 skipped (27m), py-tools 1,840 passed / 6 failed / 77 skipped, py-tests 1 failed (import)**. A **control run on plain upstream `6e38f58`** (same interpreter, same session) produces the **identical failure sets in all three steps** (0 head-only, 0 base-only; py-serve passes 19 fewer on the control — exactly this PR's 19 feature tests). None introduced by this diff; the families are the known Windows vision/subprocess ones plus a POSIX-only import in `tests.split.test_split_engine`.
- Python 3.11 minimal-serving-runtime (`requirements.txt` only): `python -m unittest serve.test_api_keys` — **19 tests, OK** (Python 3.11, `requirements.txt` only, re-run at this head)
- `tools/check --only docs` — **passed at this head** (6 checks, 0 failed, incl. the settings-register check over the config tables)
- `tests/snapshot` — the drafting round-trip snapshot tests fail identically on plain upstream on this Windows box (byte-for-byte same counts with and without this PR's commit, established on the prior head); this diff touches no launch path the snapshot suite drives differently
- **Real engine, not mock** (RTX 3090 Ti, `parallel 2`, two live clients): the startup line reports the configured keys; `/status` shows the per-key view (names, never keys); concurrent saturation streams on one capped key serialize at the gate (183 live samples on the production server, engine 0.1.62: in-flight never exceeded the key's max); a latency-sensitive client on the `reserved_capacity` key answered immediately throughout; the `api_keys` block survived repeated `setup`/update rewrites via `carry_over` (two production updates since 15 NOV)
Not run at this head: the ubuntu-24.04 leg of `py-serve` (no Linux box here) — CI covers it once the runs are approved.
## Status note
Maintainer response of 20 NOV: kept open as a product decision, not scheduled for 0.2.17; upstream remains without an implementation at 0.2.18 (`6e38f58`). Validation is kept current at every head rebase; the fork branch stays rebased on main per the running production deployment.
本站相关内容
相关页面的快捷入口。