#5950 [Engine] STRATA_COALESCE_PREFILL never co-reads on a piped layer split or for a burst reaching an idle server
open · @Maziar123 · 1 Kommentare · Auf GitHub
Server & APIMulti-GPUAMD / HIPNVIDIA / CUDAModels & quants
Beschreibung
### Area Engine (with one serve change) ### Desktop app, Terminal or source? Source build ### What happened With `STRATA_COALESCE_PREFILL=4` (`STRATA_COALESCE_TOKENS=65536`, `STRATA_COALESCE_MIN_REQS=2`, `STRATA_COALESCE_WAIT_MS=40`) on a piped layer split, the server logs ``` [strata] coalesce: 3 waiting prompt(s) (49004 tokens) read with this request's 16334 ``` but the engine never logs `co-read`: every prompt is read alone (`strata batch: read 16340 tokens on a thread of its own ... (concurrent ...)`). I traced three causes. All three paths are opt-in; the default path is not involved. 1. Engine, `STRATA_PIPED_CONCURRENT=1` on a piped split: a BGEN's read runs on the concurrent reader thread (the `piped && pr_conc` block in `src/program/generate.cpp`), which calls `sp.run` directly. `co_run` is never reached and the BCO prompts are dropped without a log line. 2. Engine, once routed past that reader: `run_multi` declines every co-read with `strata batch: co-read of 3 prompt(s) declined (prefill: multi-prompt read declined: a --prefill-pipe plan sets the chunks of this read)`. This happens because `run` applies the `STRATA_PREFILL_PIPE` plan (`force_chunk`) before `co_run`. 3. serve: the first request of a burst finds the engine idle and takes the solo `GEN` path, which has no `_coalesce`. Its neighbours arrive 6–54 ms later (measured at the gate) and wait. When the first request is promoted to a slot, `_coalesce` sends their BCO lines together with a BGEN whose prompt is already cached (`prompt 16341 tokens = 16340 reused + 1 read`). Nothing is read, so `co_run` does not run and each follower then reads alone. Proposed change (scope question before a PR): - engine (+14/−4): a BGEN that carries co-reads skips the concurrent reader, and the pipe plan applies only to a one-prompt read (`co_run`'s fallbacks included). Without BCO lines nothing changes. - serve (+9): with coalescing on, a request that finds the engine idle waits `STRATA_COALESCE_WAIT_MS` first. If a neighbour is waiting by then, it takes the batch path. With coalescing off nothing changes. Is this the direction you want for coalescing on layer splits? If yes, I will open one PR per change. ### Smallest steps or command that reproduce it Engine only (cold, 4 prompts of ~16K tokens): ``` STRATA_PREFILL_PIPE=3072 STRATA_PIPED_CONCURRENT=1 STRATA_BATCH_MTP=1 STRATA_PIPED_MTP=1 \ python3 tools/batch_test.py --exe <strata> --config <qwen config> --batch 4 --n 4 --max-new 32 --pad 16384 --skip-solo --co ``` Result: no `co-read` line, and admissions take 25.6 s, the same as without `--co` (25.5 s). Server: `python3 -m serve.server --engine strata --parallel 4` with the env above plus the coalescing keys; four 16K requests sent together. ### Reproduction status Reproduced locally (engine logs and timings below). The cause analysis was confirmed by the fixed builds. ### Strata version Source at our branch based on upstream `146df3c` (serve 0.2.12, engine 0.1.58). Built for CUDA sm_75 with CUDA 13.4 / GCC 13.3, using the pinned llama.cpp/ggml `3cf03257`. ### Graphics card, memory and system 5x Quadro RTX 8000 48 GB (sm_75), Threadripper PRO 9975WX 32 cores, 503 GiB RAM, Ubuntu 24.04 container (image 0.2.8-cuda13). ### Model, size and settings Qwen3.8-Flash-Next IQ3_S. `--layer-split auto` over 5 cards, `--batch 4`, `--kv int8 --kv-resident 32768`, `--spec 4 --mtp ...`, serve `--parallel 4`. ### Logs Results with both fixes applied: - Engine A/B: four serial reads 25.4 s → one co-read pass of 63,471 tokens in 17.5 s; admissions 18.3 s. - Server, Qwen alone at 16K, coalescing off vs on: - 4 requests: wall 32 → 26 s - 2 requests: wall 17 → 16 s - 1 request: unchanged - Engine log lines: `co-read: 1 more prompt(s) read in the same pass as this request's 16339 tokens (12259 ms)` and `co-read: 2 more prompt(s) read in the same pass as this request's 16322 tokens (15958 ms)`. Limits: - Inside a burst the first request now answers later (all four finish together). - Co-read tokens match serial layer-major reads (`STRATA_PREFILL_LAYER_MAJOR=1 STRATA_PREFILL_LM_FORCE=1 STRATA_PIPED_CONCURRENT=0`) in 3 of 4 slots, at both 16K and 1.5K prompts. One slot flips an early near-tie, so the output is not bit-exact on this card. - Measured only on CUDA sm_75 with this model; HIP and SYCL were not built.
Mehr auf der Site
Links zu Install, Modellen, Releases.