#6004 [Engine] Prefill hangs on the second prompt chunk, whatever its size (Windows, sm_120, Edge0-120B Q3_K, 0.1.55): watchdog kill 0xC0000409

open · @bbirdkt · 1 commentaires · Sur GitHub

BenchmarksServer & APINVIDIA / CUDAModels & quantsWindowsLinux

Description

### Summary

On Windows 10 + RTX 5060 Ti 16 GB (sm_120), engine 0.1.55, `Edge0-120B-q3k`:

| request | result |
| --- | --- |
| 64-token prompt | OK, ~2 s |
| **292-token prompt** | reads tokens 0..255 in 2 s, then the **36-token tail chunk** makes no progress for 60 s, the #29 watchdog stops the engine (`exit 3221226505` = `0xC0000409`), the server restarts it, the request returns HTTP 503 |

So this is not a prompt-length limit: **the second prefill chunk never gets its GPU completion**, even when that chunk is only 36 tokens. The first chunk runs at ~128 tok/s (256 tokens in 2 s); the second one does not move at all.

The same pack and the same engine build (0.1.55) on Linux + RTX 3090 24 GB reads a 116,060-token prompt in 43.6 s without a single stall, so this looks Windows / sm_120 specific.

### Environment

| | |
| --- | --- |
| Engine | `0.1.55` (`strata generate: session is up (engine 0.1.55)`) |
| App | `edge0-infer serve` (Edge0 0.2.0) |
| Model | `Edge0-120B-q3k` (86 GiB pack, `edge0-infer inspect` -> `ok: true`) |
| OS | Windows 10 Pro 19045 (10.0.19045) |
| CPU | Intel Xeon Platinum 8175M, 24 expert-pool workers |
| RAM | 127.7 GiB (page file peak 38.4 GB) |
| GPU | 2 x RTX 5060 Ti 16 GB, sm_120, driver 610.88; ran on `--device cuda:1` (GPU 0 is held by an unrelated service) |
| CUDA | 13.1 (`cublas64_13.dll`) |
| VRAM free with everything loaded | 1,706 MiB |

### Reproduction

```
edge0-infer serve --model F:\Edge0\models\Edge0-120B-q3k --device cuda:1 --context 16384
```

Engine argv as it appears in the engine's own log:

```
strata.exe --serve --pack . --native ./model-skeleton.gguf --native-dense-gguf ./model-skeleton.gguf
  --ple-gguf ./ple-fp8.gguf --ple-io direct --expert-profile ./profile.bin --expert-cache auto
  --prefill 256 --spec 4 --spec-min-p 0.5 --mtp ./mtp --max-context 16384 --kv int8
  --vram-reserve-mib 2048
```

Then send any request whose prompt tokenizes to more than 256 tokens. The text is irrelevant; this is the one used below (one Vietnamese sentence repeated 8 times; the chat template makes it 292 prompt tokens):

```
POST /v1/chat/completions
{"model":"edge0-120b","max_tokens":16,"temperature":0.2,"reasoning_budget_tokens":8,
 "messages":[{"role":"user","content":"<sentence> x8  Tra loi ngan: OK?"}]}
```

The same request with the sentence once is 64 prompt tokens and answers in ~2 s, so the same server, model and flags are healthy for short prompts.

### Where it stops (measured, `--prefill 256`)

Every full chunk is read at ~2 s per 256 tokens; the engine then dies on the chunk that follows:

| prompt tokens | chunks read OK | dies at | size of the chunk that hangs |
| --- | --- | --- | --- |
| 64 | - (single chunk) | - | works |
| **292** | 256 | **256** | **36** |
| 408 | 256 | 256 | 152 |
| 524 | 256 + 256 | 512 | 12 |
| 640 | 256 + 256 | 512 | 128 |
| 756 | 256 + 256 | 512 | 244 |
| 1,017 | 256 x 3 | 768 | 249 |
| 1,220 | 256 x 4 | 1,024 | 196 |

With `--prefill auto` or `--prefill 2048` a 1,220-token prompt becomes a single chunk and dies the same way; the progress line then never advances past `1,220 of 1,220`.

A 64-token prompt still works as the 2nd and 3rd request of a session, so this is the chunk boundary, not warm-up or the expert cache.

### Engine stall report

```
strata serve: no progress for 60 s during a request (reading the prompt (batched): waiting for the GPU
(attention, router) at layer 1 of the prompt chunk from token 256) - stopping the engine so the server
starts it again (issue #29)
strata serve: stall report (engine 0.1.55): stage "reading the prompt (batched): waiting for the GPU
(attention, router) at layer 1 of the prompt chunk from token 256" for 60 s; 0 layers served since the
last finished step (0 = stopped, more = slow)
  VRAM free when this request began: 1706 MiB
  expert pool: epoch 0, batch epoch 0: 0 of 0 jobs claimed, 0 done; 24 of 24 workers parked, 24 sleeping; mode 0
  expert pool threads: w0=sleeping w1=sleeping ... w23=sleeping; host idle for 67029 ms
  verify window (last window, not the current stage): 0 tokens at position 0, host at layer step 1;
  the GPU rang 0; flags: served 0, plan (A) 0, copies (B) 0
  memory: 31047 MiB resident, 43716 MiB committed, 27277 MiB RAM available; 9484046 page faults so far
  wrote the thread stacks to F:\Edge0\models\Edge0-120B-q3k\strata-stall-1941028.dmp (attach it to the issue)
strata: released the verify window's GPU waits (#267): the GPU finished in 0 ms
```

That last line looks like the actual bug. Once the watchdog fires and the engine releases the window's GPU
waits, **the GPU reports it had already finished, in 0 ms**. So the work did complete on the device, but the
completion never reached the host: the engine sits in `waiting for the GPU (attention, router)` at layer 1
with 0 layers served, 0 of 0 expert jobs claimed and all 24 pool workers parked.

Memory is not the cause at this point: 27 GiB RAM available, 1,706 MiB VRAM free when the request began,
no TDR (`nvlddmkm` Event 4101) anywhere in the log. `nvlddmkm` Event 153 ("Error occurred on GPUID: af00")
appears ~3 s *after* each fail-fast, so it reads as a consequence, not the cause. The GPU is idle and ~45 C
throughout (0 % util, P8/P1, 26 W).

### What does not help

Everything below was run against the same 1,220-token prompt (and 292-token for the fastest case) and still
stalls at the same chunk:

* `--prefill 256` (the repro), `--prefill auto`, `--prefill 2048`
* `--resident-experts`, `--expert-cache-per-layer`
* `--max-context` 4096 / 16384 / 262144
* `STRATA_PLE_BATCH=0` (the #224 workaround)
* `STRATA_PF_SMALL=0` (the #5966 workaround)
* `STRATA_PREFILL_CPU_SHARE=0` (the CPU share the engine turns on for prompt chunks below 1024 tokens;
  the failing chunk is exactly such a chunk, so this was the first suspect)
* `--no-capture --no-pool` -> the engine refuses to start (exit 1 before ready)

### Control: same build on Linux

Same pack, engine 0.1.55 (commit 7b60076), RTX 3090 24 GB, `--prefill auto --spec 4 --spec-min-p 0.5 --kv int8
--resident-experts --kv-resident 32768 --pool-workers 8 --max-context 262144`:

| prompt | wall | prefill |
| --- | --- | --- |
| 1,220 | 1.9 s | ~760 tok/s |
| 2,380 | 2.3 s | ~1,090 tok/s |
| 7,890 | 5.2 s | ~1,550 tok/s |
| 23,550 | 14.2 s | ~1,670 tok/s |
| 58,060 | 35.0 s | ~1,660 tok/s |
| 116,060 | 43.6 s | ~2,660 tok/s |

No stall at any length, so the prompt path itself is fine on Linux.

### Related issues

* **#29 / #2065** (the 60 s watchdog family): same message, but those are decode stalls at very low VRAM
  headroom. Here it is deterministic on prefill chunk 2 with 1.7 GiB free.
* **#5966** (`0xC0000409` while copying experts >= 256 in the `PF_SMALL` path): its workaround
  (`STRATA_PF_SMALL=0`) does not help here.
* **#224** (same watchdog family on a 5070 Ti / sm_120 with long prompts, `STRATA_PLE_BATCH=0` works):
  closed, and `STRATA_PLE_BATCH=0` does not help here, so either it regressed or this is a different path.
* **#3807** (batched-prompt hang family on 0.1.46/0.1.47): same shape, but none of the workarounds from the
  sibling reports apply here.
* **#954** (sm_120 + `STRATA_PF_FUSED`): `STRATA_PF_FUSED=0` does not help either.

I am reporting it separately because this one is reproducible on *every* long prompt, on a prompt of only
292 tokens, and none of the documented workarounds in the family change it.

### Dumps

`strata-stall-*.dmp` is written on every failure: 18 files so far in `F:\Edge0\models\Edge0-120B-q3k\`
(~8 MB total), including `strata-stall-1941028.dmp` from the 292-token run above. Happy to attach the dumps
(or a specific one) if that helps - just say which.

Sur le site

Liens install, modèles, releases.