#481 serve engine deadlocks in SleepConditionVariableSRW during a long streaming reply; stall watchdog never fires

说明

## Environment

- **Engine:** 0.1.34 ready-made release build (`BUILD.json`: version 0.1.34, source "release", portable, cuda 13.0, archs 75/86/89/120), `engine\strata.exe` timestamp 2026-10-02 00:55 UTC. Observed twice live on this exact build. The 0.1.35 changelog doesn't list anything matching this, so I assume it is still present (not yet re-tested on 0.1.35).
- **Hardware:** NVIDIA RTX 5060 Ti 16 GB (sm_120) — the only CUDA device; Intel i5-14600K (6 P-cores + 8 E-cores, no AVX-512); 64 GB RAM; Windows 11 Pro build 26200.
- **Serve:** `serve/server.py`, OpenAI-compatible endpoint on 127.0.0.1:8080.
- **Engine args** (one line for easy re-use):

```
--pack <packs\iq3_xxs> --native <shard1> --ple-gguf <shard2> --expert-profile <custom profile> --expert-cache auto --prefill auto --spec 4 --mtp <mtp\rt> --max-context 205824 --kv int8 --kv-resident 32768 --vision --vram-reserve-mib 700 --pcie-frac 0.35 --spec-min-p 0.70 --pool-workers 9 --pool-affinity auto
```

(`--pool-affinity auto` was added between the two live observations; the first deadlock predates it, so affinity is not implicated.)

- **Client:** an agentic coding client over OpenAI-compatible SSE: tool-call loops, 34K–85K-token prompts with heavy prefix reuse, `reasoning_effort: high` → multi-minute `<think>` streams, occasional client-side cancels.

## What happens

Mid-generation, the engine **deadlocks**: output stops, the GPU drops to 0% while VRAM stays allocated (15.1 GB held), the engine process's total CPU time freezes (every thread blocked), and the request never completes nor errors. The client hangs until the whole service is restarted. The engine log's last line is always a normally-completed earlier request — nothing is written at the moment of the hang, on stderr or stdout.

Happened 4+ times over two days, **only** with this agentic client; browser chat and short requests never trigger it. Both live-observed cases froze ~74 s into a `<think>` stream (once at thinking token 2459, once at 9717+).

## Evidence (second occurrence, captured live)

1. Engine process 12+ minutes into the freeze: all 30 threads in `Wait` (2× `EventPairLow`, rest `Unknown`/`UserRequest`), total CPU time identical across samples.
2. Full memory dump taken (`comsvcs MiniDump /full`), parsed without symbols. **Main thread:**

```
ntdll.dll!NtWaitForAlertByThreadId+0x14            <- RIP
ntdll.dll!RtlSleepConditionVariableSRW+0x1de
KERNELBASE.dll!SleepConditionVariableSRW+0x38
strata.exe+0x35f118                                <- caller of SleepConditionVariableSRW
```

The dump was deleted after analysis (64 GB), but I can reproduce, re-dump and pull **all 30 thread stacks** with a working minidump parser if that helps identify the lock partner.
3. **The stall watchdog never fires.** After 12+ minutes of a fully frozen process there was no `no progress for ... s ... (issue #29)` line and no `strata-stall-<pid>.dmp`. Whatever the main thread waits on, the watchdog's progress accounting does not cover it.
4. **`serve/server.py` wedges with the engine.** After I force-killed the deadlocked engine process, the server did not respawn it on the next request (the request handler stays stuck); only restarting the whole server recovers.

## Ruled out

- **RAM/VRAM exhaustion:** memtest86 clean; the wedge reproduced with ~24 GB RAM free; engine VRAM was at its planned steady state (520 MiB free, no `LOW` warning) — and the deadlock is a permanent wait, not a slowdown.
- **GPU driver TDR:** no Event 4101 / LiveKernelEvent at the freeze times.
- **The client's cancels as the direct trigger of the *freeze*** — a cancel was in flight around the first incident, but the second freeze happened with no cancel; cancelled requests complete cleanly otherwise (v0.1.34's fast cancel works).

## Frequency / repro shape

Every occurrence: long agentic session (tool calls + big reused prefixes) with `reasoning_effort: high` → a thinking stream several minutes long → engine freezes mid-stream. Never in ~50 short chat/browser requests. I cannot reproduce it on demand yet, but it recurs within a day of agentic use.

## Asks

1. Resolve `strata.exe+0x35f118` in the 0.1.34 release build (a `.pdb`/map for release builds — or a hint which `std::condition_variable::wait` in the serve path sits there — would let me self-resolve and give you the function).
2. Why the stall watchdog (`STRATA_WATCHDOG_S`) doesn't cover this wait; ideally a stuck engine should still self-report and dump.
3. `serve/server.py`: detect a dead (or force-killed) engine mid-request and respawn, instead of wedging the handler.

## Workaround I run meanwhile

An external scheduled task tails the server's stdout (Tee-Object to a file) and restarts the whole service when a `[strata] thinking:` progress line goes stale for >90 s. Turns the hang into a ~2 min hiccup.

Happy to attach: per-thread stack dump of all 30 threads on the next occurrence, `STRATA_TRACE=1` log through a freeze, or the engine + server logs around either incident.