#5787 [Engine] serve: python.exe dies natively under long agent sessions — every dump faults inside CPython on the same nested `any(genexpr)` path (`serve/frontend.py` `_has_tag`)

open · @jesse890423 · 9 评论 · 去 GitHub 看

Setup & installServer & APIAMD / HIPNVIDIA / CUDAModels & quantsWindows

说明

## Environment

- Strata Terminal **0.2.12 / build 0.1.58** (Windows), Control 0.1.58, engine 0.1.58-windows-x86_64-cuda-cuda13
- Bundled interpreter: python.org **CPython 3.12.15** — `data\terminal\0.1.58\python\python312.dll`, file version 3.12.15, PE timestamp 2026-10-09
- Windows 11 10.0.26100 x64, NVIDIA GeForce RTX 5090 32 GB, 128 GB RAM
- Launch: `--context 512k --kv int8 --vision on --parallel 1 --mtp off --drafting dflash2 --set resident-budget-gib=64 --set vram-reserve-mib=3584 --set rope-scaling=yarn --set rope-scale=2 --set yarn-orig-ctx=262144 --access lan --port <port> --no-tui`
- Model: Qwen3.8-Flash-Next GGUF UD-Q4_K_XL + DFlash2 drafter
- Disclosure: this install has two local patches to the *Control/Terminal `setup.py`* that only affect download/mirror selection and shard pre-checks. Nothing in `serve/` or the engine binary is modified.

## Summary

The **serve process (`python.exe`, the OpenAI/Anthropic API listener)** died natively four times in ~25 hours while one LAN client ran long agentic sessions. The C++ engine was unaffected every time; `host.log` only reports that the server ended on its own, with no Python traceback.

| local time | exit code | WER | context size just before |
|---|---|---|---|
| 2026-11-10 02:36 | `0xC0000409` | (no dump) | long agent session |
| 2026-11-10 11:44 | `0xC0000005` | `python.exe.17132.dmp` (mini) | ~237,684 tokens (mid tool-call write) |
| 2026-11-10 23:15 | `0xC0000409` | `python.exe.12132.dmp` (mini) | ~242,308 tokens (right after a request completed normally, `cancel=False`) |
| 2026-11-11 03:02 | `0xC0000005` | `python.exe.37844.dmp` (**full, 303 MB**) | long agent session, same client |

I analysed the dumps with cdb 10.0.26100.9457 plus the **`python312.pdb` that ships inside the Terminal package** (17.7 MB, private C/C++ symbols — thank you, that is what makes this report precise: function names, inlined frames and argument values below are exact). All three analysed crashes are memory-safety failures **inside CPython itself**, and all three land on the *same* code path.

## Evidence

Crash A — `c0000005` (read), faulting instruction is an inlined `Py_INCREF`:

```
python312!Py_INCREF [inlined in python312!PyFunction_NewWithQualName+0x5e]
    00007ff8`098ca6ae  mov eax, dword ptr [rdx]      ; refcount++ of a garbage pointer
    previous:            mov rdx, qword ptr [rcx+78h]

python312!Py_INCREF
python312!_Py_NewRef(obj = 0x91cc51fa`8f936709)      ; not a pointer at all
python312!PyFunction_NewWithQualName(
        code     = 0x00000298`882ac030,              ; plausible heap address, already freed/dirty
        globals  = 0x00000298`8827c400,
        qualname = 0x00000000`00000000)+0x5e
python312!_PyEval_EvalFrameDefault+0x3d0c
(inline) _PyEval_EvalFrame / gen_send_ex2
python312!gen_iternext(gen = 0x00000298`9237f3e0)+0x9e
python312!builtin_any(iterable = ...)+0x2a3          ; Python-level any(genexpr)
python312!_PyEval_EvalFrameDefault+0xc8d
python312!_PyFunction_Vectorcall+0x17c
python312!slot_tp_init+0x143 -> type_call -> _PyObject_MakeTpCall+0x1a5
... (53 frames) ...
python312!PyThreadState_Clear+0x23a <- ucrtbase!thread_start   ; a threading.Thread worker
```

Reading one of the "code object's" pointer fields yields `0x91cc51fa8f936709` — its low dword still looks like a heap page offset, the high dword is junk. That is a code object whose memory was already freed and reused, i.e. a use-after-free / stray 32-bit store, hit while MAKE_FUNCTION increfs it.

Crash B — `c0000409`, `Parameter[0]=0x2` = **`FAST_FAIL_STACK_COOKIE_CHECK_FAILURE`**:

```
python312!__report_gsfailure(stack_cookie = 0x1a00)+0x1d     ; int 29h
python312!_PyEval_EvalFrameDefault+0xcbe                     ; the eval loop's own frame cookie is gone
(inline) _PyEval_EvalFrame / gen_send_ex2
python312!gen_iternext+0x9e
python312!builtin_any+0x2a3
python312!_PyEval_EvalFrameDefault+0xc8d
(inline) _PyEval_EvalFrame / gen_send_ex2
python312!gen_iternext(gen = 0x000001e9`91bc2f80)+0x9e
python312!builtin_any+0x2a3
python312!_PyEval_EvalFrameDefault+0xc8d
python312!_PyFunction_Vectorcall+0x17c ...
```

Not a stack overflow (that would be `0xC00000FD`) and not `abort()`/uncaught C++ (`subcode 4`). A local buffer of the interpreter's own frame was overrun.

**A third crash landed at 03:02 the next morning, and this time I had a full (heap) dump.** Same walker, third different field destroyed:

```
python312!_PyEval_EvalFrameDefault+0xfd5    mov word ptr [r14], ax        ; spec: write specialised opcode into co_code_adaptive
python312!_PyEval_EvalFrameDefault+0xfd7    mov rax, qword ptr [rbx+8]    ; rbx->ob_type
python312!_PyEval_EvalFrameDefault+0x100b   mov rdx, qword ptr [rax+0E0h] ; tp_iternext  -> ACCESS_VIOLATION
python312!_PyEval_EvalFrameDefault+0x1012   lea rax,[python312!repeat_next]
python312!_PyEval_EvalFrameDefault+0x1019   cmp rdx,rax                   ; GET_ITER/FOR_ITER inline-cache specialisation
```

The object header is readable in a full dump:

```
0:010> dq 0x273a5f46f20 L2
00000273`a5f46f20  00000000`00000002 00007ff8`0f8c4020
                   ob_refcnt = 2      ob_type = 0x00007ff8`0f8c4020   <-- not mapped
```

`0x7ff80f8c4020` is not inside any of the 90 loaded modules (nearest below: `python312` ends `0x7ff809524000`, `_imaging_cp312` ends `0x7ff809cb1000`; nearest above: `cd_cp312` starts `0x7ff823b50000`), and it is not a stale image range either — the dump's unloaded-module list only contains four WMI DLLs far away at `0x7ff85e8c0000`/`0x7ff861b20000`. So it is a corrupted value, not a dangling module pointer.

The recursion this time is fully visible: **5 nested levels** of `builtin_any → gen_iternext → _PyEval_EvalFrameDefault`, and the bottom of the stack is `python312!thread_run+0x48 -> bootstrap -> ucrtbase!thread_start` with `method_vectorcall`/`_PyVectorcall_Call` above it — i.e. a `threading.Thread` whose target is a bound method, running the walker.

**Across the three dumps the destruction has a distinctive shape:**

| dump | destroyed slot | value | what is wrong with it |
|---|---|---|---|
| A | `code->co_name` | `0x91cc51fa8f936709` | high dword is junk, low dword is still a plausible heap page offset |
| B | eval-loop frame's `/GS` cookie | — | 8 bytes of the interpreter's own frame overwritten |
| C | `object->ob_type` | `0x00007ff80f8c4020` | high dword is right for an image pointer, low dword is junk |

In A and C exactly **one 32-bit half of an 8-byte pointer slot** was replaced while the other half stayed correct, and the replacement values look like byte counts/offsets rather than addresses (`0x0f8c4020` ≈ 249 MB, `0x91cc51fa` ≈ 2.4 GB). That is not what a wild write, an allocation failure or a GPU fault looks like; it looks like a 4-byte store landing on the wrong half of a pointer. Two samples, so this is a characterisation, not a proof.

**The common factor** is a nested `any(generator)` inside a generator body. In the shipped 0.1.58 tree the only place with that shape is `serve/frontend.py`:

```python
# serve/frontend.py:295
def _mark_deep(v, tags=LITERAL_TAGS):
    if isinstance(v, str):  return _mark(v, tags)
    if isinstance(v, dict): return {k: _mark_deep(x, tags) for k, x in v.items()}
    if isinstance(v, list): return [_mark_deep(x, tags) for x in v]
    return v

# serve/frontend.py:305
def _has_tag(v, tags=LITERAL_TAGS) -> bool:
    if isinstance(v, str):  return any(tag in v for tag in tags)
    if isinstance(v, dict): return any(_has_tag(x, tags) for x in v.values())   # line 309
    if isinstance(v, list): return any(_has_tag(x, tags) for x in v)            # line 311
    return False
```

and both are driven once per request over the entire payload (`serve/frontend.py:359`):

```python
if not _has_tag(messages, tags) and not _has_tag(tools, tags):
```

so every request walks the whole `messages` + `tools` structure, creating a generator object and an extra frame per level per node.

## What the evidence rules out

- **Not GPU / VRAM / engine.** The serve process module list contains no `torch`, `cublas`, `cudart` or `nvinfer` — only `python312`, `unicodedata`, `_regex`, `_psutil_windows`, `_ssl`/`libssl-3`/`libcrypto-3`, `_socket`, `_lzma`, `_bz2`, `_ctypes`, `_hashlib`, `_speedups`, `_queue`, `_uuid`, `nvml`, `cd_cp312` (pydantic-core), `md_cp312`, `_wmi`, `libffi-8` + system DLLs. The engine (`strata.exe`, a separate process) stayed up every time.
- **Not the commit-limit fail-fast of #1607.** No NULL/allocation-failure signature (A dereferences a junk non-null pointer, B is a /GS cookie failure), and there were only 12 threads in the process.
- **Not an injected third-party DLL** (no AV/hook modules in the module list; `amsi.dll` is a normal system load).
- **Not plain recursion depth.** The `any()` chain is 1-2 levels deep in dumps A/B and 5 levels in dump C, with the whole stack under 80 frames — far below anything that could exhaust the interpreter's datastack. So `_has_tag` looks like the **victim site** (highest allocation churn per request, so the first thing to trip over corrupted memory), not the mechanism. I am reporting the corruption, not blaming the walker.

## Reproduction

Not deterministic, and I cannot give a minimal case — that is the honest limitation. The conditions are: one long-lived client session, ~240k-token contexts, per-request payloads containing large tool results (deeply nested JSON: file trees, ASTs, tool schemas), serve process uptime 9-12 h, three hits in 25 h. Dumps A and B are `only registers, stack and portions of memory`, so they had no heap; **dump C was captured as a full heap dump (303 MB) and is available**, and I have the matching private `python312.pdb`, so I can run any cdb command against it on request (object headers, `!heap` state, frame-chain walk to Python names).

Suggestions to catch it in the act:

1. **Enable faulthandler in the serve launcher by default** (`PYTHONFAULTHANDLER=1`, or `faulthandler.enable()` at serve startup). Both codes above (`EXCEPTION_ACCESS_VIOLATION`, `STATUS_STACK_BUFFER_OVERRUN`) are caught by it, so a field crash would leave the Python traceback in `host.log` instead of nothing. This is the single highest-value change from my side — I have had to set it locally and it will make the next occurrence diagnosable.
2. Consider making `_has_tag` / `_mark_deep` iterative with an early exit, or hoisting the `_has_tag(messages)` pre-check so the whole structure is not walked twice per request. Even if the walker is only the victim, it is where every long session dies.
3. If you have a Windows box with the same build, an ASAN/`/GS`-instrumented serve plus a replay of a large agent transcript would likely reproduce it faster than field reports.

## Questions

- Is the Windows serve layer known to corrupt memory on long agent sessions, or is there a build/flag combination you would like tested?
- 0.2.13's notes say Windows keeps Terminal at 0.2.12 on `strata update`, so serve-side fixes currently cannot reach Windows installs. Is moving Terminal on Windows planned for the next release?
- I have since enabled WER full dumps for `python.exe` and `PYTHONFAULTHANDLER=1` on the serve launcher, so the next occurrence should leave both a Python traceback in the host log and a heap-complete dump. Three dumps are available now (2 x 24 MB mini, 1 x 303 MB full) plus `guard.log`/`host.log` excerpts — tell me which cdb commands would help and I will run them and attach the output.

本站相关内容

相关页面的快捷入口。