#465 Feature request: serve concurrent requests (cross-request batching)

closed · @Viachtik · 3 commentaires · Sur GitHub

Setup & install Server & API NVIDIA / CUDA Models & quants

Description

With several clients on one GPU (orchestrator agent + sub-agents, a chat bot) everything queues: `/v1/status` reports `concurrency.serving: 1`, and DETAILS.md lists "one request at a time" as a v1 limit.

Request: batch decode steps across a few concurrent requests (even 2-4 slots), like llama.cpp `--parallel`.

Setup: RTX 3090 24 GB + 62 GB RAM, IQ3_S, 262K ctx, OpenCode multi-agent pipeline. Thanks!

Sur le site

Install Strata
Model sizes & RAM
Release notes

Liens install, modèles, releases.