反馈 / #1994
#1994 Batch slots return HTTP 400 "BGEN: no such free slot" instead of queueing; the affected conversation stays broken
open · @jsilve12 · 0 评论 · 去 GitHub 看
Server & APIMulti-GPUNVIDIA / CUDAModels & quantsDocumentation
说明
Title: Batch slots return HTTP 400 "BGEN: no such free slot" instead of queueing; the affected conversation stays broken
Body:
## Summary
With `--batch 4` (and `--batch-groups auto`) on a layer split, a request that
arrives when all slots are busy is rejected with HTTP 400 rather than waiting:
```
[strata] the engine reported an error: BGEN: no such free slot (--batch 4) or a bad max_new
[strata] 400 invalid request: BGEN: no such free slot (--batch 4) or a bad max_new
```
docs/BATCHING.md says the opposite for this case:
> "**More requests than slots** wait for a free one"
So the documented behaviour is to queue. What we observe is a hard 400.
## Why this matters
A 400 is not retryable from the client's point of view, and in our run the
affected conversation did not recover: every subsequent turn from that stream
returned the same error immediately (0 tokens, ~0.1 s). One rejection
permanently killed that stream for the rest of the session.
## Reproduction
4 concurrent HTTP clients, each holding a growing conversation (5 turns each),
against the lane while normal traffic is also in flight:
- 4 streams x 5 turns = 20 requests
- 19 succeeded, 1 stream died at turn 3 with the error above and never recovered
- turns 4 and 5 of that stream: `got: EMPTY`, `finish: ERROR:{'type': 'invalid_request_error', 'message': 'BGEN: no such "`
Each turn appends to the conversation, so prompts grow ~12k tokens per turn
(13.7k -> 25.8k -> 37.8k -> 49.9k -> 62.0k).
The conversation cache was working for the other streams
(`reused` climbing 13,663 -> 25,770 -> 37,834 -> 49,883), so this is not a
cache miss - it looks like slot exhaustion.
## Environment
- engine 0.1.42
- 2x NVIDIA CMP 170HX (GA100, sm_80), 64 GB each, layer split 24, `--trim-stage-weights`
- `--batch 4 --batch-groups auto --expert-cache 30000 --prefill auto:32768`
- `--spec 4 --mtp ... --max-context 262144 --kv fp16 --kv-resident 4194304`
- `--conversation-cache-mib 16384 --conversation-cache-slots 8`
- `/v1/status` reported `concurrency: {"serving": 4, "requested": 4}`
- `/metrics` at the time showed `live.slots` all idle and `waiting: 1`, `outside_slots: 2`
## What I think should happen
A request that cannot get a slot should wait (as documented), or at minimum
return a retryable status (503 + Retry-After) rather than 400 - and a
conversation that was rejected should be able to retry successfully on its next
turn rather than being permanently broken.
## Possibly related
- #1604 (`--batch-groups` disables the per-slot conversation cache)
- #1118 (batch slots stall when a long prompt is read while another slot decodes)
Happy to run any instrumentation you want - `/metrics` is very informative and I
can capture slot occupancy over time if that helps.
本站相关内容
相关页面的快捷入口。