#5950 [Engine] STRATA_COALESCE_PREFILL never co-reads on a piped layer split or for a burst reaching an idle server

open · @Maziar123 · 1 comments · View on GitHub

Server & APIMulti-GPUAMD / HIPNVIDIA / CUDAModels & quants

Description

### Area
Engine (with one serve change)

### Desktop app, Terminal or source?
Source build

### What happened
With `STRATA_COALESCE_PREFILL=4` (`STRATA_COALESCE_TOKENS=65536`, `STRATA_COALESCE_MIN_REQS=2`, `STRATA_COALESCE_WAIT_MS=40`) on a piped layer split, the server logs

```
[strata] coalesce: 3 waiting prompt(s) (49004 tokens) read with this request's 16334
```

but the engine never logs `co-read`: every prompt is read alone (`strata batch: read 16340 tokens on a thread of its own ... (concurrent ...)`). I traced three causes. All three paths are opt-in; the default path is not involved.

1. Engine, `STRATA_PIPED_CONCURRENT=1` on a piped split: a BGEN's read runs on the concurrent reader thread (the `piped && pr_conc` block in `src/program/generate.cpp`), which calls `sp.run` directly. `co_run` is never reached and the BCO prompts are dropped without a log line.
2. Engine, once routed past that reader: `run_multi` declines every co-read with
   `strata batch: co-read of 3 prompt(s) declined (prefill: multi-prompt read declined: a --prefill-pipe plan sets the chunks of this read)`.
   This happens because `run` applies the `STRATA_PREFILL_PIPE` plan (`force_chunk`) before `co_run`.
3. serve: the first request of a burst finds the engine idle and takes the solo `GEN` path, which has no `_coalesce`. Its neighbours arrive 6–54 ms later (measured at the gate) and wait. When the first request is promoted to a slot, `_coalesce` sends their BCO lines together with a BGEN whose prompt is already cached (`prompt 16341 tokens = 16340 reused + 1 read`). Nothing is read, so `co_run` does not run and each follower then reads alone.

Proposed change (scope question before a PR):
- engine (+14/−4): a BGEN that carries co-reads skips the concurrent reader, and the pipe plan applies only to a one-prompt read (`co_run`'s fallbacks included). Without BCO lines nothing changes.
- serve (+9): with coalescing on, a request that finds the engine idle waits `STRATA_COALESCE_WAIT_MS` first. If a neighbour is waiting by then, it takes the batch path. With coalescing off nothing changes.

Is this the direction you want for coalescing on layer splits? If yes, I will open one PR per change.

### Smallest steps or command that reproduce it
Engine only (cold, 4 prompts of ~16K tokens):

```
STRATA_PREFILL_PIPE=3072 STRATA_PIPED_CONCURRENT=1 STRATA_BATCH_MTP=1 STRATA_PIPED_MTP=1 \
python3 tools/batch_test.py --exe <strata> --config <qwen config> --batch 4 --n 4 --max-new 32 --pad 16384 --skip-solo --co
```

Result: no `co-read` line, and admissions take 25.6 s, the same as without `--co` (25.5 s).

Server: `python3 -m serve.server --engine strata --parallel 4` with the env above plus the coalescing keys; four 16K requests sent together.

### Reproduction status
Reproduced locally (engine logs and timings below). The cause analysis was confirmed by the fixed builds.

### Strata version
Source at our branch based on upstream `146df3c` (serve 0.2.12, engine 0.1.58). Built for CUDA sm_75 with CUDA 13.4 / GCC 13.3, using the pinned llama.cpp/ggml `3cf03257`.

### Graphics card, memory and system
5x Quadro RTX 8000 48 GB (sm_75), Threadripper PRO 9975WX 32 cores, 503 GiB RAM, Ubuntu 24.04 container (image 0.2.8-cuda13).

### Model, size and settings
Qwen3.8-Flash-Next IQ3_S. `--layer-split auto` over 5 cards, `--batch 4`, `--kv int8 --kv-resident 32768`, `--spec 4 --mtp ...`, serve `--parallel 4`.

### Logs
Results with both fixes applied:
- Engine A/B: four serial reads 25.4 s → one co-read pass of 63,471 tokens in 17.5 s; admissions 18.3 s.
- Server, Qwen alone at 16K, coalescing off vs on:
  - 4 requests: wall 32 → 26 s
  - 2 requests: wall 17 → 16 s
  - 1 request: unchanged
- Engine log lines:
  `co-read: 1 more prompt(s) read in the same pass as this request's 16339 tokens (12259 ms)` and
  `co-read: 2 more prompt(s) read in the same pass as this request's 16322 tokens (15958 ms)`.

Limits:
- Inside a burst the first request now answers later (all four finish together).
- Co-read tokens match serial layer-major reads (`STRATA_PREFILL_LAYER_MAJOR=1 STRATA_PREFILL_LM_FORCE=1 STRATA_PIPED_CONCURRENT=0`) in 3 of 4 slots, at both 16K and 1.5K prompts. One slot flips an early near-tie, so the output is not bit-exact on this card.
- Measured only on CUDA sm_75 with this model; HIP and SYCL were not built.

Related on this site

Quick links to related pages.