Pull requests / #2109

#2109 --batch: request priority (a free slot and the next prompt read go to the highest priority waiting)

open · @brocktice · 0 comments · View on GitHub

Server & APINVIDIA / CUDAModels & quantsDocumentation

Description

## Why

We serve Qwen3.8-Flash-Next IQ3_S on two RTX PRO 6000 (one engine per card, `parallel: 4`, `--batch-mtp`, all experts in VRAM) to several clients at once: an interactive coding agent, a log-alert summarizer that must answer within seconds, and an evaluation battery that sends thousands of requests a day.

With first-come-first-served waiting, a burst of battery requests filled all 8 slots and the alert summarizer queued behind it, so its calls timed out and our monitoring paged us. More slots don't fix it, because a burst can always fill them. Under SGLang we used `--enable-priority-scheduling` (a per-request `priority`), and we lost that when we moved to Strata.

We've been running the same idea in a router in front of Strata for several days. The alert summarizer's slowest call over a full day was 4.6 s (1,210 calls) while the battery kept 8 requests in flight. This PR moves it into the server, where it can see the slots and the #656 give-way path.

## What it does

`"priority"` in the request body is an integer, higher goes first. It's the same field SGLang and vLLM read, so gateways like LiteLLM already pass it through. If the field is absent or not an integer, it counts as 0.

With `--batch`:

- **A free slot** goes to the highest-priority request waiting for one.
- **The control lines** (the next prompt read or admission) go to the highest-priority request waiting for them.
- **#656 give-way:** a long read gives way to a waiting request of a *higher* priority, whatever its length. It never gives way to a much shorter request of a *lower* priority. Within one priority level, the old rule applies (a prompt under half as long). `YIELDS_MAX` still caps how often a read gives way.
- **No preemption:** a running request is never paused.

With every request at 0, all checks reduce to the existing code paths.

**Deadlock note:** requests waiting for a *slot* never hold back the *control lines*. A read that gave way holds a slot and needs the lines to continue and free that slot, whatever its priority; otherwise long/long/short with mixed priorities would deadlock (the ENGINE_REVIEW finding 1 shape). A request that gave way and is letting others go first (`after_epoch`) also doesn't outrank anyone until its epoch passes.

**Not covered:** the single-request path without `--batch` (the server's `fifo` lock). That lock is shared with the vision encoder and settings changes, so changing it seemed like a separate discussion.

## Tests

New tests in `serve/test_parallel.py`, using the existing fake engine:

- `test_a_free_slot_goes_to_the_highest_priority_waiting`: both slots busy, a priority-0 request waits, then a priority-5 one arrives; the first free slot goes to the priority-5 request.
- `test_a_long_read_gives_way_to_a_higher_priority`: two prompts of the same length; the later, higher-priority one makes the first read give way.
- `test_no_way_given_to_a_lower_priority`: a short priority-0 prompt does not make a priority-5 read give way.
- `Priority.test_request_priority` and `test_outranked_skips_a_request_letting_others_go_first` (unit).

All three service tests fail on `main` (checked three times each) and pass with the change. `serve/test_parallel.py` passed 5 runs in a row. The full `serve/` suite passes except `test_responses.NoLeakedSampler.test_server_close_ends_the_sampler`, which also fails on unmodified `main` in 2 of 3 runs here (a sampler thread from an earlier test is still shutting down).

**Real engine:** v0.1.43 on one RTX PRO 6000 (Blackwell), Qwen3.8-Flash-Next IQ3_S, `parallel: 4`, `--batch-mtp`. The same request mix was run with and without the field:

| Scenario | Without priority | With priority |
| --- | --- | --- |
| 4 slots decoding 700-token stories, 8 more waiting (priority 10); a short request arrives 2 s in. First token: | 45.3 s / 39.1 s (10 and 8 of the waiting requests went first) | 12.2 s / 12.0 s (the first slot to free, nobody ahead of it) |
| A ~84K-token read under way (priority 10); an equally long request arrives 3 s in. Its first token: | 29.5 s (waited for the whole first read) | 16.0 s (the first read gave way at 16,384 tokens, parked in slot 2, and finished at 33.3 s) |

No errors; every request finished.

`docs/BATCHING.md` has a short paragraph describing the behavior.

Happy to change the shape: a config default per API key, `/metrics` waiting counts per priority, or the non-batch path.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Related on strata.com

Editorial links to help you install, pick models, or read release notes — not part of the upstream thread.