Issues / #3464

#3464 [Other] one OpenAI-compatible address in front of several providers, next to the local server

open · @Buddychek · 1 コメント · GitHub で見る

needs-format

本文

**Problem.** People running Strata next to a hosted model have no single address for both. A client either talks to the local server or to a provider, and if one is down the client fails. The only workaround today is to run two clients and pick by hand in the application code.

**Proposed behaviour.** `tools/busbar_gateway.py` (already written, stdlib only, no GPU and no engine) listens on one OpenAI-compatible `/v1` address. A name like `fast` stands for a pool of members with weights: 80 of 100 requests go to Claude, 20 to GPT, spread evenly by smooth weighted round robin rather than in clumps. A member that fails is tried on the next member, then on the pools named in `failover`, and is put on cooldown after `max_fails` failures in a row. Anthropic answers are converted to the OpenAI shape, SSE streams pass through. `--plan` prints the routing for the next N requests and sends nothing.

**Tradeoffs.** It is a separate tool, not part of the Strata server: nothing in the default path changes, and the gateway needs no engine. It is single-process and holds no provider keys beyond the config and environment. It does not retry a member more than once per request, and it does not limit request body size.

**Evidence so far.** 34 unit tests run without network or provider keys (`python -m unittest tools.test_busbar_gateway`), covering the exact split, the spread, failover, cooldown, both wires, the Anthropic-to-OpenAI conversion, streaming in both directions, and the HTTP routes. Code is in `tools/busbar_gateway.py`, `tools/busbar.example.json` and `docs/BUSBAR.md` on the branch `tools/busbar-gateway`.

**Ask.** Is a standalone gateway in `tools/` in scope for this repository, or should it live as a separate project? If it is in scope, I will open the pull request against this issue.

関連リンク

インストール・モデル・リリースへのサイト内リンク。