#2049 Deterministic deadlock at the completion handshake after the last block: host at layer step 48 with the GPU already rang - 5/5 stalls identical, on 0.1.41 and 0.1.43 (RTX 4080 / Windows / IQ3_XXS native pack / 256K)

Descrição

## Summary

The `STRATA_WATCHDOG_S` watchdog (#29) fires every time, because the request makes **no host-side progress for the full 60 s**. But the stall is **not** slow I/O, **not** memory pressure and **not** a version regression. All five occurrences freeze at **exactly the same point** - the handshake after the model's last block:

```
verify window (last window, not the current stage): 6 tokens at position 88089,
    host at layer step 48; the GPU rang 48; flags: served 48, plan (A) 48, copies (B) 48
  expert pool: epoch 439406, batch epoch 439406: 18 of 18 jobs claimed, 18 done; 5 of 5 workers parked, 5 sleeping; mode 0
  expert pool threads: w0=sleeping w1=sleeping w2=sleeping w3=sleeping w4=sleeping; host idle for 61509 ms
```

This model has **48 blocks** (`blk.0` .. `blk.47` in the pack's `index.txt`), so **`layer step 48` is one past the last layer**. At that moment:

- the GPU has already rung 48 (`served 48, plan (A) 48, copies (B) 48`)
- the expert pool has **nothing in flight** (18 jobs claimed, 18 done, `epoch == batch epoch`)
- all 5 pool workers are `parked` / `sleeping`
- yet the host stays blocked for 61 s and never returns

The whole forward pass has completed on the device and the host simply never gets past step 48.

## Not a 0.1.43 regression

| # | engine | engine started | context position | host idle |
|---|---|---|---:|---|
| 1 | 0.1.41 | 2026-10-10 08:43:08 | 183,045 | 61,657 ms |
| 2 | 0.1.41 | 2026-10-10 14:43:49 | 107,264 | 61,478 ms |
| 3 | 0.1.41 | 2026-10-10 14:51:44 | 121,757 | 61,306 ms |
| 4 | 0.1.43 | 2026-10-12 12:21:31 | 88,089 | 61,491 ms |
| 5 | 0.1.43 | 2026-10-12 12:48:42 | 106,983 | 61,509 ms |

Identical signature in all five, across two releases. **Downgrading does not help** - 0.1.41 shows it three times.

On Windows the manifestation is Application Error 1000 on `strata.exe`, exception `0xc0000409` (the report code for `abort()` / `__fastfail`), same faulting offset `0x52f519` in both 0.1.43 occurrences.

## Ruled out

- **Slow I/O (#1407)**: the `no step finished for %d s, but the file tier is still being read` branch never fired - `grep -c "no step finished"` is **0** across the whole log, so `STRATA_WATCHDOG_IO_S` is not involved.
- **Memory pressure**: `committed` is 56,158-57,912 MiB against a 63.2 GiB guard with 16,150-19,527 MiB still available; `resident` actually **fell** from 30.3 GiB (occurrence 4) to 16.8 GiB (occurrence 5).
- **The NaN guard (#879) added in 0.1.43**: `non-finite logits` never appears in the log.
- **A single unlucky run**: 5/5 identical, and occurrences 4 and 5 were 8 minutes apart within one session.

## Environment

```
Engine   : 0.1.43 (also reproduced on 0.1.41)
OS       : Windows 11
GPU      : NVIDIA RTX 4080 (sm_89, 16 GB; display on iGPU)
CPU      : AMD Ryzen 7 7800X3D
RAM      : 64 GB
Model    : Swift-Qwen3.8-Flash-Next-GSQ-RCO-IQ3_XXS (native IQ3_XXS pack), 48 blocks
Launch   : --max-context 262144 --kv int8 --mmap-experts --spec 6 --mtp <path> --spec-min-p 0.7
           --pool-workers 5 --resident-budget-gib 40 --pcie-frac 0.5 --vram-reserve-mib 768
           --kv-resident 20480 --conversation-cache-mib 8192 --conversation-cache-slots 4
           --conversation-cache-min-free-mib 1024
```

No non-default value the watchdog could be sensitive to; this is the documented 256K configuration.

## Reproducibility

Every occurrence needs a **long** session. Short A/B segments (a few minutes each) never hit it. On 0.1.43 the first appeared ~25 min into a session and the second 8 min after the automatic restart. The trigger therefore looks tied to long sessions at large context positions (88K-183K) rather than to any single request shape.

## Related

- **#1234** - same stall class reported on 0.1.39 / RTX 4090 (`stage "decode -1"`). Ours freezes at `stage "request -1"`, and the new information here is that the site is **deterministic and identical across versions**.
- **#29** - the watchdog itself. Its design is working as intended here (it turns a permanent hang into a 60 s hiccup plus an automatic restart), so this report is about **what it is catching**, not a request to turn it off.
- **DETAILS.md** ("Generation stops mid-answer...") asks for the thread stacks and says *"attach both"*. Both parts are attached: five `strata-stall-<pid>.dmp` files, and one text file with all five stall reports.

## Attachments

- `E:\AI\Strata\strata-stall-13600.dmp` (359,712 B)
- `E:\AI\Strata\strata-stall-22336.dmp` (355,289 B)
- `E:\AI\Strata\strata-stall-23088.dmp` (355,289 B)
- `E:\AI\Strata\strata-stall-27724.dmp` (357,605 B)
- `E:\AI\Strata\strata-stall-30728.dmp` (362,028 B)
- `_issue_strata_stall_excerpt.txt` - all five stall reports in one file

[strata-stall-13600.dmp](https://github.com/user-attachments/files/33320964/strata-stall-13600.dmp)
[strata-stall-22336.dmp](https://github.com/user-attachments/files/33320961/strata-stall-22336.dmp)
[strata-stall-23088.dmp](https://github.com/user-attachments/files/33320962/strata-stall-23088.dmp)
[strata-stall-27724.dmp](https://github.com/user-attachments/files/33320965/strata-stall-27724.dmp)
[strata-stall-30728.dmp](https://github.com/user-attachments/files/33320963/strata-stall-30728.dmp)

[_issue_strata_stall_excerpt.txt](https://github.com/user-attachments/files/33320967/_issue_strata_stall_excerpt.txt)