Issues / #465
#465 Feature request: serve concurrent requests (cross-request batching)
closed · @Viachtik · 3 コメント · GitHub で見る
Setup & installServer & APINVIDIA / CUDAModels & quants
本文
With several clients on one GPU (orchestrator agent + sub-agents, a chat bot) everything queues: `/v1/status` reports `concurrency.serving: 1`, and DETAILS.md lists "one request at a time" as a v1 limit. Request: batch decode steps across a few concurrent requests (even 2-4 slots), like llama.cpp `--parallel`. Setup: RTX 3090 24 GB + 62 GB RAM, IQ3_S, 262K ctx, OpenCode multi-agent pipeline. Thanks!
関連リンク
インストール・モデル・リリースへのサイト内リンク。