#3382 [Engine] STRATA_PREFILL_CPU_SHARE_BATCH=1 (opt-in): the prompt's CPU share with --batch, for a read while no slot decodes; short reads -21% to -31%

Descrição

## Problem and result

The prompt path's CPU share (`STRATA_PREFILL_CPU_SHARE`) is off whenever `--batch` is set: generate.cpp sets `share_pool = o.batch <= 0 && !o.no_pool`, so with `"parallel": 2` the variable has no effect. The server runs with batch slots for concurrency, so a short read there always waits on the GPU alone: below the full-streaming chunk size (`stream_all_min`, here 2048) a chunk copies only its routed experts over PCIe, and on this machine that wait is about half of a 1-2K-token read.

With `STRATA_PREFILL_CPU_SHARE_BATCH=1` the pool is handed to the prompt path as it is without batch slots, and before each read the serve loop checks the slots: a read while any slot is decoding takes no share (the slots' verify windows use the pool), a read with none does. The existing CPU-share rules then decide as before (which chunks, how many experts).

Result on 2x RX 6900 XT (below), 10 interleaved cold-start pairs: real-text reads under 2,048 tokens -28.8% (median), a 1,733-token one -20.9%, conversation continuations adding 455/781 tokens -30.7%, a 34K read -2.3% (its last chunk, 1,378 tokens); reads of 2,048 tokens and more unchanged. Without the variable the default path is byte-identical.

## Related discussion

None found. I could not find a stated reason for `batch <= 0` in the code or history; this keeps it as the default and only adds the opt-in.

## Approach

- generate.cpp: `share_pool = (o.batch <= 0 || share_batch) && !o.no_pool`, with one line at load when the variable applies; a `share_gate()` before each read (in `read_part`'s run and before the concurrent reader starts) calls `Prefill::set_share_blocked(busy > 0)` from the slots' `active` flags, and prints the decision for the first 4 reads.
- prefill.cpp: `set_share_blocked()` sets an atomic that the chunk's existing pool-hold test also checks (`!g_share_blocked`).
- The share still needs the CPU share itself enabled (`STRATA_PREFILL_CPU_SHARE=auto` or `1`; on a layer split it is not on by default).

## Validation

Environment: 2x RX 6900 XT (gfx1030, 16 GB, PCIe 4.0 x8 each), Ryzen 5 5600X (6 cores), 128 GB DDR4-3200, Ubuntu 26.04 (kernel 7.0.0), ROCm 10.0.0. Revision: main e88ee39d (v0.2.0-beta + 12 commits) and this branch on it. Built with setup.py's HIP options (`-DSTRATA_PREFILL_MMQ=ON -DSTRATA_Q6K_EXPERTS=ON -DSTRATA_MMQ_KQUANTS=ON`, gfx1030, the pinned ggml, Release). Model: Qwen3.8-Flash-Next GSQ-RCO IQ3_S, layer split over both cards.

**Passed:**

1. Default path byte-identical: deterministic config (`parallel` 1, no pipeline windows, `STRATA_PREFILL_CPU_SHARE=0`, `STRATA_LOGPOS` top-256 log-probabilities), 8 chats of real text, 4,067 scored positions: main and this branch without the variable `cmp` identical.
2. Speed: `parallel` 2 (`--batch 2`), `--pipeline-windows 2 --batch-mtp`, `STRATA_PREFILL_CPU_SHARE=auto`, `STRATA_PREFILL_STREAM_MIN=2048`, `STRATA_HC_PERSIST=1`, `STRATA_HIP_PROMPT_F16=1`, `STRATA_SH_STREAM=1`, a rocBLAS table via `STRATA_ROCBLAS_TUNING`. The same binary with and without the variable, 20 cold starts alternating A B A B; each arm sends the same 33 requests one after another (so no slot is decoding during a read): 27 single real-text prompts of 430-3,521 tokens (English prose, Chinese prose, C++), then a 34K-token conversation and 5 continuations of it adding 455-3,168 tokens. `prompt_ms` from /metrics. Median over the 10 pairs of each group's mean change:

   | reads | median change | faster |
   |---|---|---|
   | 15 single prompts under 2,048 tokens (430-1,542) | -28.8% (pairs -28.6 to -29.0) | 10/10 |
   | 1 single prompt of 1,733 tokens | -20.9% | 10/10 |
   | 11 single prompts of 2,048+ tokens | -0.0% (pairs -0.2 to +0.1) | - |
   | 34,146-token first read (4 chunks of 8,192 + one of 1,378) | -2.3% | 10/10 |
   | 2 continuations adding 455 / 781 tokens | -30.7% | 10/10 |
   | 3 continuations adding 2,136-3,168 tokens | +0.0% | - |

   e.g. a 430-token Chinese prompt 1,722 -> 1,146 ms, a 455-token continuation 1,722 -> 1,168 ms. The `=1` arms printed `CPU share for this read: on (0 slot(s) decoding)`, the others no such line; 0 stalls.
3. `ctest` (STRATA_BUILD_TESTS=ON; the engine binary is the same one, sha256 unchanged): 137 tests, 131 passed. The 6 failures are the same 6 on main e88ee39d built the same way: `ple_parity` (needs a Q2_0 GGUF this machine does not have), `hip_prefill_mmq_parity` (SEGFAULT on main too), `native_hadamard_test`, `tq_cpu_parity`, `tq_cpu_parity_avx2` ("this ggml has no trellis types"), `isa_guard_check` (`tq_rows_avx512` not listed). The branch adds no test: the change is a scheduling condition; item 2 checks it end to end (the per-read confirmation line).

4. CUDA compile only (no NVIDIA GPU here): CUDA 13.2.2 (nvcc 13.2.86) with GCC 15.2, `-DSTRATA_ENABLE_CUDA=ON -DSTRATA_Q6K_EXPERTS=ON -DSTRATA_MMQ_KQUANTS=ON -DCMAKE_CUDA_ARCHITECTURES=86`: the engine builds; the one warning in generate.cpp (line 11576, an unused variable) is there on main too, none from the changed lines.

**Failed:** none observed.

**Not run:**
- Reads beside a decoding slot on this revision: by design they take no share; I measured that two concurrent requests are unchanged on 0.1.45 with this patch, not on 0.2.0.
- Output quality with the share on: the CPU computes some experts, so the bits differ from the GPU-only read and depend on timing, as with the CPU share without batch slots today; I did not measure KL for it on this revision.
- CUDA execution (compiled only, item 4), the SYCL and Metal builds (no toolkits here; SYCL builds its own copy of prefill.cpp, Metal compiles this one, and the new `Prefill::set_share_blocked` is defined in it, so no stub is needed - not verified by a Mac build), a single-GPU setup, Windows, other GPUs.

## Limits

- Opt-in; it only lifts the `batch <= 0` condition, the CPU-share rules themselves are unchanged.
- The gate is taken when a read starts: a slot that begins decoding during a long read does not take the pool back from it.
- The gain is for reads below the streaming chunk size on a machine where those wait on PCIe (here 2x x8, 45% of the experts in VRAM); a box with the experts resident would see little.

Developed with an AI coding assistant; every number above was measured on the machine described here.

🤖 Generated with [Claude Code](https://claude.com/claude-code)