#1762 SYCL `--batch 2`: a prompt read in parts while another slot decodes leaves non-finite residuals (prompt-path cache-slot loan), unless `--no-prefill-borrow`

Description

> Strata 0.1.40.3 (`d5ea713`) and main (`fb58e0d`, 0.1.41), SYCL port, AOT `bmg-g31`. Arc Pro B65 32 GB (PCI `8086:e222`,
> IP 20.2.0), xe driver, PCIe Gen4 x16, Debian 13 LXC + Docker (`strata-sycl-dev` built from Intel's apt oneAPI 2026.1.1
> and compute-runtime 26.27.39122.11 / IGC 2.38.2 / Level Zero 1.32). Qwen3.8-Flash-Next **Coder IQ1_M**, one GPU,
> `--stream-experts --spec 4 --mtp --kv int8 --max-context 131072 --prefill 4096 --kv-resident 16384 --vram-reserve-mib 1400`,
> `"parallel": 2` (all 12288 experts resident; the two slot sessions take 0.58 GiB each). Launcher defaults
> `STRATA_VERIFY_DEVICE_PLAN=1 STRATA_VERIFY_NO_HOST=1`.

- Client A: short prompt, streaming 400 tokens. Client B, 3 s later: ~12K-token prompt, streaming 400 tokens
  (`batch-repro.py`). The prompt path borrows 1036 cache slots (1.98 GiB); B's prompt is read in 3 parts with slot 0
  decoding between them (`the prompt was read in 3 parts, the slots decoding 8468 ms between them`).
- `STRATA_DBG_NAN=1`: `prompt end: last residual row 8 non-finite (max |x| 3.29e+38)` for B's prompt.
- Afterwards: `captured the batch window over slots 0,1`, then `over slots 1`; ~400 windows run (`avg 1.73 rows,
  62.71 ms/window ... emit 0.00`) but neither client receives tokens (A stuck at ~100 tokens, B at 1).
- With `--no-prefill-borrow` the non-finite residuals are gone (`last residual row 0 non-finite` on every prompt end),
  at a cost: the expert cache drops to 10,826 slots and B's prompt reads at 77 tok/s instead of ~550. (The run still
  stalls: issue 2.)
- Single slot (no `parallel`) never shows it: the loan only overlaps decoding when batch slots interleave a prompt.

Guess: a borrowed cache slot (holding prompt buffers) is read by the other slot's batch window before the loan is
returned/refilled.

<details><summary>Engine log, 0.1.41 + STRATA_DBG_NAN=1 (excerpt)</summary>

```
strata generate: session is up (engine 0.1.41)
strata verify: batch windows of up to 2 sequences (layers [0, 48))
strata batch: slot 0 takes 153 tokens (copied in 132.7 ms)
strata verify: captured the batch window over slots 0
strata dbg: prompt end: last residual row 8 non-finite (max |x| 3.29e+38); PLE history 0 non-finite (max |x| 15.3); GDN state 0 0 non-finite (max |x| 3.06)
strata verify: captured the batch window over slots 0
strata dbg: prompt end: last residual row 8 non-finite (max |x| 3.29e+38); PLE history 0 non-finite (max |x| 12.5); GDN state 0 0 non-finite (max |x| 3.09)
strata dbg: prompt end: last residual row 0 non-finite (max |x| 5.36); PLE history 0 non-finite (max |x| 14); GDN state 0 0 non-finite (max |x| 2.92)
strata batch: the prompt was read in 3 parts, the slots decoding 53757 ms between them
strata batch: slot 1 takes 11932 tokens (copied in 176.4 ms)
strata serve: prompt 11932 tokens = 0 reused + 11932 read in 203051 ms (58.8 tok/s), 1 generated in 32 ms (30.8 tok/s), drafts accepted 0 of 0, 1 checkpoints
strata verify: captured the batch window over slots 0,1
strata verify: captured the batch window over slots 1
strata batch: 419 windows, avg 1.68 rows, 167.56 ms/window = run 165.27 (CUDA0 GPU-reach wait 0.00 + CPU experts 0.00) + commit 2.29 + emit 0.00; per layer-window: CPU experts 0.00, VRAM hits 0.00, PCIe 0.00; 3.2 rows/s over 219713 ms of wall time (admissions included)
```
</details>

<details><summary>Engine log, 0.1.41 + --no-prefill-borrow + STRATA_DBG_NAN=1 (excerpt)</summary>

```
strata generate: expert cache 10826 slots, 20.63 GiB of VRAM; policy is
strata generate: session is up (engine 0.1.41)
strata verify: batch windows of up to 2 sequences (layers [0, 48))
strata batch: slot 0 takes 173 tokens (copied in 134.8 ms)
strata verify: captured the batch window over slots 0
strata dbg: prompt end: last residual row 0 non-finite (max |x| 5.87); PLE history 0 non-finite (max |x| 15.3); GDN state 0 0 non-finite (max |x| 3.06)
strata batch: 283 windows, avg 1.00 rows, 35.75 ms/window = run 34.37 (CUDA0 GPU-reach wait 27.50 + CPU experts 1.75) + commit 1.37 + emit 0.01; per layer-window: CPU experts 0.35, VRAM hits 9.62, PCIe 0.03; 4.7 rows/s over 59956 ms of wall time (admissions included)
strata dbg: prompt end: last residual row 0 non-finite (max |x| 5.81); PLE history 0 non-finite (max |x| 12.5); GDN state 0 0 non-finite (max |x| 3.09)
strata dbg: prompt end: last residual row 0 non-finite (max |x| 4.89); PLE history 0 non-finite (max |x| 14); GDN state 0 0 non-finite (max |x| 2.92)
strata batch: the prompt was read in 2 parts, the slots decoding 10061 ms between them
strata batch: slot 1 takes 11932 tokens (copied in 192.9 ms)
strata serve: prompt 11932 tokens = 0 reused + 11932 read in 155677 ms (76.6 tok/s), 1 generated in 36 ms (27.7 tok/s), drafts accepted 0 of 0, 1 checkpoints
strata verify: captured the batch window over slots 1
strata batch: 399 windows, avg 1.00 rows, 37.73 ms/window = run 36.34 (CUDA0 GPU-reach wait 28.12 + CPU experts 3.36) + commit 1.39 + emit 0.01; per layer-window: CPU experts 0.63, VRAM hits 9.28, PCIe 0.09; 26.5 rows/s over 15055 ms of wall time (admissions included)
```
</details>

<details><summary>Repro: batch-repro.py</summary>

```python
#!/usr/bin/env python3
"""batch-repro.py: reproduce the Strata batch-slot hang with direct API calls (no OpenCode).

Client A: short prompt, 400 tokens. Client B, 3 s later: ~8K-token prompt (forces the prompt path's cache-slot
loan seen right before both hangs), 400 tokens. Streams both; a client is HUNG when no token arrives for STALL s.
Usage: batch-repro.py [base_url] [stall_s]   (default http://localhost:8080, 180)
Prints one line per client: done/hung, tokens, seconds, tok/s; exit 0 = both done, 1 = a hang.
"""
import json, sys, threading, time, urllib.request

BASE = sys.argv[1] if len(sys.argv) > 1 else "http://localhost:8080"
STALL = float(sys.argv[2]) if len(sys.argv) > 2 else 180.0
filler = "\n".join(f"line {i}: the quick brown fox jumps over the lazy dog near the river bank" for i in range(1, 600))
PROMPTS = {
    "A": "Write a Python function that parses ISO dates, with docstring and three unit tests.",
    "B": filler + "\n\nSummarise the text above in five bullet points, then write a haiku about foxes.",
}
results = {}

def run(name):
    body = json.dumps({"model": "x", "max_tokens": 400, "stream": True, "reasoning_effort": "low",
                       "messages": [{"role": "user", "content": PROMPTS[name]}]}).encode()
    req = urllib.request.Request(BASE + "/v1/chat/completions", body, {"Content-Type": "application/json"})
    t0 = time.time(); last = t0; n = 0; state = "done"
    try:
        with urllib.request.urlopen(req, timeout=STALL) as r:
            for raw in r:
                line = raw.decode("utf-8", "replace").strip()
                if not line.startswith("data:") or line == "data: [DONE]": continue
                d = json.loads(line[5:])
                delta = (d.get("choices") or [{}])[0].get("delta", {})
                if delta.get("content") or delta.get("reasoning_content"): n += 1; last = time.time()
    except Exception as e:
        state = f"hung/err ({type(e).__name__}: {str(e)[:60]})"
    dt = time.time() - t0
    results[name] = (state, n, dt, n / max(dt, 1e-9), time.time() - last)

ta = threading.Thread(target=run, args=("A",)); tb = threading.Thread(target=run, args=("B",))
ta.start(); time.sleep(3); tb.start(); ta.join(); tb.join()
bad = False
for k in "AB":
    state, n, dt, tps, quiet = results[k]
    print(f"client {k}: {state}, {n} tokens, {dt:.1f} s, {tps:.1f} tok/s, quiet for the last {quiet:.0f} s")
    bad |= state != "done"
sys.exit(1 if bad else 0)
```
</details>