#2892 [Bug]: unbatched replicas can return zero tokens and cross-request DONE statistics

Description

### What happened

Two single-GPU, unbatched replicas can return an empty/zero-token HTTP reply even though the native engine generated the correct tokens. Adjacent successful replies can also receive another request's DONE statistics (prompt/decode time, generated count, cached tokens).

Fix proposal and complete investigation: **#2886**. This issue records the bug independently from the implementation PR.

Observed while testing KVGROW, but both KVGROW and resident-KV unbatched replicas use the affected Python wrapper. This is not evidence of an elastic-KV arithmetic, model-quality, or GPU-driver failure. The native engine generated 12 valid tokens while the API returned 0.

**Expected:** after EOS, a stop string or disconnection, finish this request's native STOP/DONE cleanup before another request owns the same pipe; report only this caller's own DONE statistics. Two different replicas should still progress independently.

**Actual historical evidence:**

| Synthetic case | Prompt | API output | Native row whose timing was borrowed | Corrected API / own DONE |
| --- | ---: | --- | --- | --- |
| SLOT-1-RVN-8020 |4165|0 tokens, finish=length|4167 prompt /13 generated|12 correct tokens, stop; native 4165/12|
| SLOT-3-RVN-23858 |4167|13 correct tokens|4165 prompt /12 generated|13 correct tokens; native4167/13|

Payload bytes and SHA256s are identical before/after. The historical `RVN` marker is only a synthetic conversation identifier; the actual model is **GSQ-RCO IQ3_S**. [Requests, original/fixed SSE and native DONE evidence](https://github.com/dynaxbill/Strata/blob/f8527faeccbd195e2a1bbc369c0847c071cc8c5c/tests/data/replica_lifetime/gpu-historical-cases.json).

### Two independent root causes

1. **Pipe lifetime:** `ReplicaEngine.generate` forwards its child with `for ...: yield`, but its old `finally` releases `run_lock` before explicitly closing that child. A waiting successor can consume the predecessor's unread DONE and terminate without a token; the predecessor's delayed child cleanup can consume the successor's tokens/DONE. Cleanup errors can also be lost as ignored generator-finalizer exceptions.
2. **Statistics ownership:** closing/draining the child under the lock is necessary but insufficient. Unbatched native `last` is shared. After unlocking, another caller can overwrite it before `Service` reads the completed request's statistics. A caller-local snapshot must be captured before unlock, while preserving the old identity when no new DONE arrived.

The original GPU failure did not record a per-token thread trace. The timing association is unique within rounding tolerance; source inspection and event-controlled replay independently establish the two mechanisms.

### Reproduction without a model or GPU

With Strata's Python dependencies installed:

```sh
git clone --branch fix/replica-done-ownership-20261017 https://github.com/dynaxbill/Strata.git
cd Strata
python -m unittest -v serve.test_replica_lifetime
python tools/replay_replica_lifetime.py --output ./replica-repro-new
```

Choose an output directory that does not exist. The replayer uses baseline commit 477ac13cbe8e582c20484871ce459d20d82df09d; fetch it in shallow checkouts or supply `--baseline-file`.

The 18 contracts import the repository's **actual StrataEngine reader/pump and ReplicaEngine**; only the native producer is simulated. Two tests use real OS pipes. Events force a legal handoff instead of hoping to win a scheduler race. The replayer never edits the checkout.

| Control | Passed/18 | Interpretation |
| --- | ---: | --- |
| Original upstream |9|Lifetime, cross-read, statistics and cleanup-propagation regressions reproduced|
| Close-only |16|Pipe handoff repaired; caller-statistics and real-pipe statistics assertions still fail|
| A: close + caller-local snapshot |18|Both independent failures repaired in these contracts|

The same counts were obtained on Windows CPython 3.12.14 and Linux CPython 3.12.3. Original/close-only red logs are intentional controls. [Replayer](https://github.com/dynaxbill/Strata/blob/f8527faeccbd195e2a1bbc369c0847c071cc8c5c/tools/replay_replica_lifetime.py) / [logs and manifest](https://github.com/dynaxbill/Strata/blob/f8527faeccbd195e2a1bbc369c0847c071cc8c5c/tests/data/replica_lifetime).

### Three repairs evaluated / selection

- **A — selected in #2886:** explicit child close/drain plus a new DONE snapshot into caller-local `last`, before releasing the existing per-replica lock. No new worker/queue or global serialization; native protocol/math and defaults unchanged.
- **B — offline prototype:** a request-scoped `NativeTurn` owns the lock, child iterator and result; `yield from` delegates close and idempotent cleanup snapshots statistics before release. 18/18 offline contracts, GPU/API NOT_RUN; a larger object/iterator refactor.
- **C — offline prototype:** one native pipe-owner worker per replica with bounded request-specific token/result channels and backpressure.18/18 offline contracts, GPU/API NOT_RUN; extra handoffs and incomplete restart/unload/overload/watchdog integration.

A was chosen for its smaller serving change. Earlier CPU-only real-pipe microbenchmark first fixture-token p50/p99 at65 input IDs: A 0.132/0.191ms, B 0.139/0.204ms, C 0.428/0.671ms, original 0.143/0.227ms;1CPU/768MiB,13 symbolic output IDs,8 rotated rounds. These are **not LLM TTFT/GPU TPS**, have no confidence intervals, and do not establish an A/B model speedup. [Full alternative comparison](https://github.com/dynaxbill/Strata/blob/f8527faeccbd195e2a1bbc369c0847c071cc8c5c/docs/replica_request_lifetime.md#three-evaluated-repairs-and-why-a-was-selected).

### Strata version, GPU, OS

- Real GPU qualification: Strata engine **0.1.44.1**, source `296b59d2f73c8f04a805df29c2fa97655adace78` plus exact repair A.
- Current-main CPU/serving check: `477ac13cbe8e582c20484871ce459d20d82df09d` plus A. `replicas.py` is unchanged between those upstream bases. Newer native code in current main was **not** GPU-qualified.
- **2x RTX 3090 24GiB/SM86**, physical **2x EPYC 7663**; VM **64vCPUs,2 NUMA nodes,384GiB assigned RAM**, Ubuntu 26.04 LTS VM host; NVIDIA driver **610.43.02**.
- Container Python **3.12.3**, CUDA toolkit **13.0.48**, libcudart/cuBLAS13; GGML/llama.cpp pin `3cf03257f219afbe7334045ff7c6a06ac68c627d`. [Actual dependency/build/link metadata](https://github.com/dynaxbill/Strata/blob/f8527faeccbd195e2a1bbc369c0847c071cc8c5c/tests/data/replica_lifetime/dependencies.json).
- Model: Qwen3.8-Flash-Next **GSQ-RCO IQ3_S**, native/PLE two GGUF shards; exact Hub revision not captured.
- Two tested replica profiles: one native decoder/card, workers31/card, prefill8192, context262144, KVint8, spec4/minp0.5, per-card cache16384MiB/4slots. Resident uses `--kv-resident32768`; KVGROW removes it and uses INIT16384/STEP32768/HOLD1. Not an isolated full-VRAM KVGROW ON/OFF comparison.

### Engine log

These two original native rows were associated with the wrong API replies:

```text
strata serve: prompt 4167 tokens = 0 reused + 4167 read in 2210 ms (1885.5 tok/s), 13 generated in 95 ms (136.7 tok/s), drafts accepted 10 of 10, 2 checkpoints
strata serve: prompt 4165 tokens = 0 reused + 4165 read in 2199 ms (1894.2 tok/s), 12 generated in 83 ms (144.6 tok/s), drafts accepted 10 of 10, 2 checkpoints
```

The corrected same-byte SLOT-1 request now has 12 correct API tokens and its own native 4165/12 DONE; SLOT-3 has its own 4167/13 DONE. Full timing fields and source hashes are in the linked evidence.

### Completed validation and limits

- Current-main Linux: **186 selected existing serving tests +18 new contracts =204 green tests**. Existing modules run in separate interpreters; not the entire repository suite.
- Real GPU resident/KVGROW: history20/20 each; **120 completed lifecycle replies without empty outputs**, plus **8 deliberate disconnects** and correct successors; backend API9/9 per profile.
- Both GPUs grew KV to262144. Two distinct260088-input/2048-output requests, cold1+warm3 rounds: **8/8 exact outputs**, both replicas busy, output intervals overlap2. Post-growth history20/20.
- Patched image in the original layer-split C4 production profile: **B0 11/11**, history20/20,4 native decoding slots observed, backend9/9, authenticated API10/10 and separate client-host nonce PASS.
- Native binary unchanged: SHA256 `4ea6a2fa3361ec22d764944a570dd76ec03106295687e17b343798a18e106ff4`. Fixed replicas.py SHA256 `76b68ca40ebf0aa26f974cc66351322843f3bc4d77275c781f0f511639a03fe1`.

The earlier pinned-image combined183-test run had one existing sampler-thread-count failure (182/183); its assertion was retained and excerpt published. Isolated modules passed183/183, Responses40/40; original combined control183/183. Its unique scheduling trigger was not established.

KVGROW260K warm3 measured114.54 combined E2E tok/s and max-TTFT median3.183s, cached260081/request. Layer C1 short was76.82 ->74.58 E2E tok/s (-2.92%), one epoch each; that layer shape does not exercise this wrapper, so this is not a causal speed claim. B0 context262144/slack8 is not a shape-matched Dominik comparison. B0 TTFT starts after headers; the separate benchmark starts before submission.

NOT_RUN: GPU execution of current main, HIP/SYCL/Metal hardware, full-head KL/perplexity, long soak, physical native crash/missing-ACK/restart/unload/reboot recovery, fullKV/GROW0 and replicas C4 with A. KVGROW startup's463/464MiB CUDA-free warning remains a headroom limitation, not a passing long-soak claim. Only A received GPU/API qualification.

[Detailed investigation](https://github.com/dynaxbill/Strata/blob/f8527faeccbd195e2a1bbc369c0847c071cc8c5c/docs/replica_request_lifetime.md) / [per-request results](https://github.com/dynaxbill/Strata/blob/f8527faeccbd195e2a1bbc369c0847c071cc8c5c/tests/data/replica_lifetime/gpu-qualification.json). Public artifacts are synthetic and have a SHA256 manifest; credentials, home paths and private host addresses are excluded. Full private1171-file archive hash is recorded for provenance, not presented as a public download.