Pull requests / #3466
#3466 [Other] tools: the busbar gateway, one OpenAI address in front of several providers
open · draft · @Buddychek · 0 comentários · No GitHub
Setup & installServer & APIAMD / HIPNVIDIA / CUDAWindows
Descrição
**Problem and result.** A client that has both a local Strata server and a hosted model has no single address for the two. `tools/busbar_gateway.py` puts one OpenAI-compatible `/v1` address in front of several providers: a name like `fast` is a pool of members with weights, 80/100 requests to Claude and 20/100 to GPT, spread evenly by smooth weighted round robin. A member that fails is tried on the next member of the pool, then on the pools listed in `failover`, and is put on cooldown after `max_fails` failures in a row. Anthropic answers are converted to the OpenAI shape, SSE streams pass through, and `/v1/models`, `/v1/stats` and `/v1/health` report the pools. Nothing in the default path changes: the gateway is opt-in, stdlib only, and needs no engine, GPU or model. **Related discussion:** #3464 (scope not yet agreed by a maintainer - hence the draft). **Approach.** The split has to be exact, so the router uses the same smooth weighted round robin nginx uses instead of a random draw; `--plan` shows the next N routes without sending anything, so a user can check a config before pointing a client at it. The last commit in this branch is the hardening a live PC forced: urllib hands a socket aborted before a status line as a raw `ConnectionAbortedError` (WinError 10053), which the first version did not catch, so an upstream that dropped the connection killed the request with a traceback instead of failing over. Now every call path catches `OSError` and `http.client` errors, a stream the provider cuts short is counted as a failure, and a client that stops reading is not blamed on the member. **Validation** (Windows 11, Python 3.14.7, commit `c109e625`, no GPU, no provider key, no network): - Passed: `python -m unittest tools.test_busbar_gateway` - 34 tests, `OK`, 15 runs in a row. - Passed: the same suite before this commit failed the streaming tests in 4 of 6 runs with `ConnectionAbortedError [WinError 10053]`; after it, 0 of 15. The cause was measured, not guessed: a stand-in upstream that returns headers and a body and then closes without flushing aborts the connection about once in 60 attempts on this PC (60-run A/B: 59 OK without the flush, 60 OK with it). - Passed: `python tools/busbar_gateway.py --config tools/busbar.example.json --plan --requests 20` - `counts: claude=16, gpt=4`, the exact 80/20 spread. `--plan --pool nope` prints the known pools and exits 2. - Passed: streaming end to end over the real HTTP handler (a urllib client POSTs `/v1/chat/completions` with `stream: true` and reads the SSE back). - Failed: none. - Not run: the serve suite and the CUDA/HIP/SYCL builds - no engine, serve or setup file is touched, so nothing there can change. The real providers are not called: no API keys on this machine, so the upstream calls themselves are only covered by stand-in members. **Limits.** One process, no request body size limit, one attempt per member per request, and the round-robin state is per pool, so two pools asking for the same member do not share a counter. No tests were run against Anthropic, OpenAI or Bedrock.
No site
Links install, modelos, releases.