#2275 [Direction] Aim Strata at long-context multi-user: a peer's prompt read leaves a decoding slot at 3.67 tok/s, the batch window emits 1 token/slot, and 4 streams take 4.04x a single read to their first tokens (2x2080 Ti layer split, 96K)

Description

### This is a direction request, not a bug report

We would like **Strata aimed at long-context multi-user serving**: several 96K–192K conversations on one engine, all of them making progress, each keeping a usable tok/s, and nobody waiting for a peer's whole prompt to be read. Today that is not the case on a 2-GPU layer split, and the two gaps we measured look **structural rather than tuning**: they are the two core assumptions of the current design, and we are willing to test whatever you think should change first.

### What we are asking for (design direction)

1. **Make "long context + multiple users on a layer split" a first-class target**, not a single-GPU-only feature. Today the concurrency work (drafting inside the batch, a read that gives way) exists on one GPU and is refused or unreachable on a split.
2. **Per-slot drafting on a layer split.** The batch window emits **1 token per slot per window**; the solo path emits 2–5 (it has the drafts). A wide speculative row set that works *inside* the pipelined groups (PR #1524's design, extended past `stages.empty()`) is the biggest lever we found.
3. **A prompt read that can give way cheaply**, and/or a **prefill that takes more than one sequence per pass**. Today one read at a time is the rule (`server.py`: *"one prompt read at a time"*), so n streams cost n(n+1)/2 read times to reach their first tokens (4 streams at 96K: **412 s**).

### Summary of the evidence

On a 2×RTX 2080 Ti layer-split rig running Qwen3.8-Flash-Next IQ3_XXS at 96K–192K context we measured the multi-user path end to end. Two losses dominate:

1. **A peer's prompt read takes the whole machine for its duration.** A request that has already produced its first token gets **0 windows** while another prompt is read: measured **3.67 tok/s** over the 63 s of the peer's read (vs 36.6 tok/s for the same slot when it is not competing), and with 4 streams the first-token times are 51.1 / 102.9 / 154.6 / 206.4 s.
2. **On a layer split the batch window cannot use the drafts**, so it emits **1 token per window** (33 ms) while the solo path emits 2–5 (7.4 ms/token): a **4.5×** per-stream loss that no configuration we found can address.

We are not asking for a patch. We are asking whether "long-context multi-user on a layer split" is a direction you would take, and we offer to run any A/B on this rig.

### Rig and flags

- 2× RTX 2080 Ti 11 GB (sm_75), Linux, 123 GB RAM, **no swap**; CPU pool 22 threads
- Engine v0.1.43, Qwen3.8-Flash-Next GSQ-RCO IQ3_XXS (2-part GGUF; the 320,001,536-row PLE table is 28.8 GB)
- layer split auto → CUDA0 layers 0–23 / CUDA1 24–47, expert caches 9 400 + 6 790 slots
- `--batch 4 --batch-groups auto` → **2 groups of 2 slots**; `--batch-mtp` refused: *"it is not run with helper expert caches on a layer split"*; `--remote-expert-opt`; `--pipeline-windows 2` (off: "not with --batch slots")
- `--kv int8 --kv-resident 65536`, `--max-context 262144`; prompt chunk auto **26 624 tokens** (384-slot ring); PLE `direct` (startup probe 69 943–75 741 rows/s, "fine")

### What we measured

**Single stream (references)**: 96K prefill **1 555 tok/s**, decode **~135 tok/s** (drafts accepted 147/179); 192K prefill 1 600 tok/s, decode 121 tok/s.

**Two streams, 96K, MT=256** (unique nonces ⇒ `reused 0`):

| | request A (admitted first) | request B |
|---|---|---|
| TTFT | 50.4 s | 102.2 s |
| end-to-end | 109.2 s | 109.2 s |
| decode while the *other* read ran | **3.67 tok/s** | — |
| decode in the final 7 s (both slots decoding, no reads) | — | **36.6 tok/s** |

**Four streams, 96K, MT=64**: TTFT 51.1 / 102.9 / 154.6 / 206.4 s (sum **412 s**, the reads are fully serialized), all four finish at 209.4 s.

**Where a window's time goes** (`STRATA_DECODE_TIMING=1`): 20.35–43.65 ms/window = `verify 20.33–37.98` (`GPU-reach wait 8.11–8.28` + `per-layer host 0.76–1.37` [`CPU 3.64–5.71`] + `stage 0.01–0.09`) + `commit/emit 0.01`; per layer-window `CPU experts 0.94–1.50`, `VRAM hits 8.23–9.04`, `PCIe 0.02–0.27`. **`stage` is 0.01–0.09 ms, so we do not have the #2038 launch-serialization regression** — we checked before reporting this.

**dmon (106 s, 1 Hz)**: SM mean **68.1 % / 67.1 %**, **26 % / 24 % of samples ≤ 5 %**, PWR mean **209 W / 204 W** (max 305/300; TDP 250). "Only one card above 20 %" in **52 of 106** seconds. There is GPU headroom; the contended resource looks like the per-window host/PCIe/expert orchestration.

### Why the two losses look structural

1. **`batch_read_piece()` returns 0 for the pipelined groups** (`include/strata/program/batch_read.hpp:24`), so on our route the read is one `sp.run` and — because of the early `if (piece <= 0) return sp.run(...)` in `src/program/generate.cpp` (~line 10907) — **the decode-share block and the BYIELD point below it are unreachable**. `STRATA_BATCH_DECODE_SHARE` is therefore inert *by construction* here (we verified: 0.5 / 1.0 / 3.0 all give the same 2.0 s of slot decoding per read on the one route where it is reachable, and 0 on ours).
2. **`--batch-mtp` cannot run on a split** (`generate.cpp:4362–4374`): it needs `--batch ≥ 2`, `--mtp`, `--spec ≥ 2`, its own GPU per stage, **not** `--batch-groups > 1`, and **not** helper expert caches / `--remote-expert-opt`. We hit the last two. Your 2×A4000 numbers (x4 aggregate 85.8 pipelined without batch-mtp vs 64.1 serial with it) say the pipelined groups are the right trade — which is why we are *not* asking to "just enable batch-mtp"; the batch window needs a way to draft *inside* the pipelined groups.

### We ported PR #2050 to our tree and measured it

The mechanism works: with `STRATA_PIPED_READ_PIECE=26624`, the log shows `a read right after a read: the slots decoded 253 rows in 8398 ms first`, and 2-chunk pieces read at 1 589 tok/s (no read-speed loss).

But on **our** rig it does not pay, and the reason is a number we would like to understand better:

| config | TTFT sum | total wall | already-decoding stream |
|---|---|---|---|
| default (one run) | **152.6 s** | **109.2 s** | 109.2 s |
| pieces = 1 chunk (26 624 tok) | 224.5 s (+47 %) | 157.9 s (+45 %) | **80.9 s (−26 %)** |
| pieces = 2 chunks (53 248 tok) | 194.9 s (+28 %) | 138.7 s (+27 %) | 105.0 s (−4 %) |
| pieces = 4 chunks (> prompt) | 153.1 s | 108.7 s | 108.7 s (control: identical to default) |

The granularity is forced: a piece must be a **whole prompt chunk**, and our chunk is **26 624 tokens** (sized by the expert cache: 29 952 is refused because CUDA1's 6 790 slots cannot hold the borrow). One piece is therefore ~21 s of reading, so "giving way once" costs the joiner ~20 s. On rigs whose chunk is a few thousand tokens the same switch is nearly free.

### What we ruled out on this rig (so it is not a config report)

- `--pcie-frac 0.55` vs the auto refinement (0.17 / 0.32): identical (walls 104.9 / 109.6 s, 8.8 rows/s both); the two-card alternation is unchanged (51 % vs 49 % of seconds)
- `--kv-resident 131072` (KV block VRAM hit 52 % → **96.67 %**): decode **unchanged** (2.4 rows/s), read slightly faster
- `--batch 2` ("2 groups of 1 slot") and `--batch 4 --batch-groups 4` ("4 groups of 1 slot"): identical TTFT/wall to `--batch 4 --batch-groups auto`
- `--batch-groups 1`, removing `--pipeline-windows 2`: identical
- `--mmap-experts` (resident arena off): RSS 60.8 → 48.4 GB but read −13 % and decode −17 % — a memory trade, not a speed one
- PLE: `--ple-inflight 1024` (1 564.7 tok/s), `--ple-row-cache 8M` (1 558.1), `--no-ple-prefetch` (1 556.1), `--ple-io mmap` (1 396–1 541) vs default 1 554; `--ple-delay-us 200` **and 5000** had **no effect at all** (5 000 µs/row would be 240 ms/token, so we suspect the injector does not engage on this build — worth a look, it would be a useful instrument)
- admission round-robin (`STRATA_ADMISSION_RR=1`, our own local experiment): it does make the two first tokens arrive together, but the sum of first-token times goes 153 s → 238 s (2 chunks) / 290 s (1 chunk)

### The asks (design direction, not a patch request)

1. **Per-slot drafting on a layer split.** The batch window is 1 token/slot while the solo path is 2–5. A "wide speculative row set" that works *inside* the pipelined groups (your PR #1524's design, but not single-GPU-only) is the single biggest lever we found: it is the 4.5× per-stream gap above, i.e. one batch window carrying 2–5 tokens per slot instead of 1.
2. **Interleaving that does not drain the pipeline**, or a chunk size that can be small without losing prefill speed. #2050's cost on our rig is entirely the pipeline refill at each piece boundary (and the coarse minimum piece).
3. **Prefill that can take more than one sequence per pass.** `Prefill::run(tokens, n, pos0)` is one contiguous range and one `SessionState`; with 2 streams the TTFT sum is already 2.0× the single read and with 4 streams 4.04× (412 s). Making it n× instead of n(n+1)/2 is the fairness half of the problem; we have a costed change list for our tree if you want to compare approaches.

We can run any A/B you specify on this rig (sm_75, 2×11 GB, long context, IQ3_XXS) and report numbers in your format. We are not expecting all three; we would like to know which one you consider the real bottleneck.

### Related issues and why they do not cover this rig

- **#1906** — the same stop with `--batch-groups` on 2×3080 20 GB; **our 3.67 tok/s matches its 3.7–13 tok/s**, so this is not rig-specific. #2050 is its fix and is opt-in.
- **#2050** — same mechanism, measured above on a rig where the minimum piece is 21 s.
- **#1369** — two 32K prompts on one 5090: overlap did not beat back-to-back (single GPU, default decode share).
- **#1603** — two long-context sessions can hang (`running: 0 / queued: 0`); we hit a sibling of this once when we tried to force pieces with the normal admission (a request produced no tokens). Worth knowing: it reproduces when `batch_read_piece` is made to return >0 on the pipelined route without #2050's `pump`/`pipe_drain` pairing.
- **#1524** — the throughput design we would most like to see extended to a split; its own docs say "No AMD/multi-GPU runtime coverage is claimed".
- **#2038** / **#1413** — we checked both against this rig: `stage` is 0.01–0.09 ms (no launch serialization), and our groups are "2 groups of 2 slots" (not the "1 group of N" trap).

Thanks — the single-stream path on this rig is genuinely good (1 555 tok/s prefill, 135 tok/s decode at 96K, reproducible across our A/B legs; your interleave harness passes on the legs where it applies), which is why the multi-user gap is worth writing up.