反馈 / #1809
#1809 [Feature]: a queue cap that answers 429 + Retry-After instead of queueing without end
open · @streber42 · 0 评论 · 去 GitHub 看
说明
### What do you want to do? Run Strata behind a proxy (litellm) that sends it more requests than it can serve, and have Strata say "busy, come back in N s" once its queue is long enough, instead of queueing every request without end. Today every request waits its turn, however many there are. On two RTX 3090s (Unsloth UD-Q4_K_XL, 0.1.40.1, `"parallel": 4`) with live agent traffic (about 16K prompt tokens and 240 output tokens per request), the queue reached about 22 requests and one 45K-token prompt waited more than 6 minutes. Clients time out while they wait, and a proxy in front cannot tell "busy" from "slow", so it cannot send the overflow to another server. The workaround today is to limit concurrency in the proxy, e.g. litellm's `max_parallel_requests`. That has to be tuned per box, it knows nothing about Strata's slots, and it does not cover clients that talk to Strata directly. What would help: a setting such as `"max_queue": N`, so that a request beyond the running slots + N gets an immediate HTTP 429 with a `Retry-After` header. A 429 is what litellm and the OpenAI and Anthropic SDKs already retry, and litellm also uses it to fail over to another deployment. A 503 would make litellm treat the server as broken. The default would stay "no limit". PR #1808 implements this; it runs on that box now.
本站相关内容
相关页面的快捷入口。