Issues / #1763
#1763 SYCL `--batch 2`: after a long-prompt request joins, the first slot leaves the batch window and both requests stop emitting (with or without `--no-prefill-borrow`)
closed · @leadgtr7 · 2 comments · View on GitHub
BenchmarksSetup & installServer & APINVIDIA / CUDAModels & quantsWindows
Description
> Same setup as #1762 (0.1.40.3 and 0.1.41 `fb58e0d`, Arc Pro B65, IQ1_M Coder, `"parallel": 2`, all experts resident).
- Same two-client run. Right after `batch: slot 1 takes 11932 tokens`, the engine logs
`verify: captured the batch window over slots 0,1` and then `verify: captured the batch window over slots 1`.
Slot 0 (client A) never re-enters a batch window: frozen at 97-117 tokens. Slot 1 (client B) stays at 1 token.
- Windows keep running (402-419 windows, avg 1.0-1.75 rows, `emit 0.00/0.01`); the server shows
`thinking: 1 of max 400 tokens, 0.0 tok/s` and `thinking: 117 of max 400 tokens, 0.3 tok/s` for 10+ minutes.
- Reproduced on 0.1.40.3 and on main 0.1.41, with and without `--no-prefill-borrow` (so not issue 1's NaN), with the
expert cache 100% resident (GPU-reach wait 0.00). Earlier, with 548 experts mirrored in RAM over a Gen3 x4 link,
the same stall happened after 179 windows.
- `d9621d1` ("long, long, short under parallel: 2 deadlocked") is in 0.1.41 and does not cover it.
- Related but different: #1603 (CUDA, Windows, `parallel: 2` with two long sessions: the request vanishes from the
bookkeeping, `running: 0 / queued: 0`, engine idle). Here the request stays counted and the engine keeps running
batch windows that emit nothing.
- Single slot: fine (hundreds of agent jobs today).
Happy to run builds or env switches (`STRATA_SPIN_MAX`, debug env) on this card; the B65 is otherwise idle at night.
Logs are in the collapsible sections below; the repro script is in #1762 (companion issue).
<details><summary>Engine log, 0.1.40.3 (excerpt)</summary>
```
strata generate: --batch: 2 slot sessions on CUDA0 (0.58 GiB each); 26.34 GiB free
strata generate: --batch 2: the slot sessions take 1.16 GiB of VRAM on CUDA0 that the expert cache would otherwise hold
strata generate: expert cache 12288 slots, 23.42 GiB of VRAM; policy is
strata generate: session is up (engine 0.1.40.3)
strata verify: batch windows of up to 2 sequences (layers [0, 48))
strata batch: slot 0 takes 157 tokens (copied in 132.8 ms)
strata verify: captured the batch window over slots 0
strata verify: captured the batch window over slots 0
strata batch: the prompt was read in 3 parts, the slots decoding 8468 ms between them
strata batch: slot 1 takes 11932 tokens (copied in 180.0 ms)
strata serve: prompt 11932 tokens = 0 reused + 11932 read in 26108 ms (457.0 tok/s), 1 generated in 32 ms (31.1 tok/s), drafts accepted 0 of 0, 1 checkpoints
strata verify: captured the batch window over slots 0,1
strata verify: captured the batch window over slots 1
strata batch: 403 windows, avg 1.73 rows, 62.71 ms/window = run 60.35 (CUDA0 GPU-reach wait 0.00 + CPU experts 0.00) + commit 2.36 + emit 0.00; per layer-window: CPU experts 0.00, VRAM hits 0.00, PCIe 0.00; 16.2 rows/s over 43125 ms of wall time (admissions included)
```
</details>
<details><summary>Engine log, 0.1.41 fb58e0d (excerpt)</summary>
```
strata generate: --batch: 2 slot sessions on CUDA0 (0.58 GiB each); 26.34 GiB free
strata generate: --batch 2: the slot sessions take 1.16 GiB of VRAM on CUDA0 that the expert cache would otherwise hold
strata generate: expert cache 12288 slots, 23.42 GiB of VRAM; policy is
strata generate: session is up (engine 0.1.41)
strata verify: batch windows of up to 2 sequences (layers [0, 48))
strata batch: slot 0 takes 153 tokens (copied in 132.7 ms)
strata verify: captured the batch window over slots 0
strata verify: captured the batch window over slots 0
strata batch: the prompt was read in 3 parts, the slots decoding 5677 ms between them
strata batch: slot 1 takes 11932 tokens (copied in 178.1 ms)
strata serve: prompt 11932 tokens = 0 reused + 11932 read in 21736 ms (548.9 tok/s), 1 generated in 32 ms (31.1 tok/s), drafts accepted 0 of 0, 1 checkpoints
strata verify: captured the batch window over slots 0,1
strata verify: captured the batch window over slots 1
strata batch: 402 windows, avg 1.75 rows, 55.95 ms/window = run 53.57 (CUDA0 GPU-reach wait 0.00 + CPU experts 0.00) + commit 2.37 + emit 0.00; per layer-window: CPU experts 0.00, VRAM hits 0.00, PCIe 0.00; 18.1 rows/s over 38763 ms of wall time (admissions included)
```
</details>
Related on strata.com
Editorial links to help you install, pick models, or read release notes — not part of the upstream thread.