#6227 [Engine] Windows CUDA 0.1.60: a start stalls at the prompt-path init (no `strata hc` line), holds 23 GB of VRAM and the engine cannot be terminated

open · @fiestorra · 1 comments · View on GitHub

Server & APINVIDIA / CUDAModels & quantsWindows

Description

### Area

`[Engine]  (speed, kernels, models, the server and API)`

### Desktop app, Terminal or source?

Terminal (the `strata` command)

### What happened

Two symptoms on the same machine with the same engine (0.1.60, windows-x86_64-cuda-cuda13). The first is the #29 family. The second is what this report is about: **a stall during the start, before READY, that writes no stall report at all, keeps ~23 GB of VRAM and cannot be terminated.**

**A. A start stalls at the prompt-path init.**

The profile runs with lazy loading and `idle_unload_s: 150`, so after a couple of hours of agent traffic (loads, 150 s idle unloads, reloads) a reload stalled. The engine log stops at:

```
strata serve: the prompt path borrows 1870 CUDA0 cache slots (3.07 GiB)
```

A healthy start prints this immediately after that line:

```
strata hc: CUDA0: the hyper-connection read runs as staged (the norm per token and stream, the down projection's activations staged ahead by cp.async); checked bit for bit against the plain read on this card (STRATA_HC_SPLIT=1 or 0 for the earlier ones)
```

In the stalled start that line never came. Observed while it lasted:

- one thread (`strata/0`) at ~95% CPU for more than 10 minutes, GPU utilisation 0%, no disk I/O, no new log line;
- no `strata-stall-*.dmp` and no stall report: the #29 watchdog does not cover the start;
- at `engine_ready_s` (946 s) the server printed `[strata] the engine did not start (try 1 of 3); again in 15 s` and spawned another engine, which stalled at the same line (`try 2 of 3`, then the third attempt).

**B. The stalled engine cannot be terminated, and Windows cannot get the GPU back.**

When the start was aborted (`strata runtime stop`, then `Stop-Process -Force` on the engine):

- `tasklist` still listed `strata.exe` (24.8 GB working set) while `taskkill /F /PID` answered *"There is no running instance of the task"*, and it stayed like that for over 10 minutes;
- the card kept 23,154 MiB of dedicated memory (WDDM `GPU Adapter Memory` counter) with no other client visible;
- `nvidia-smi` and NVML hung until the device was restarted;
- Windows logged no TDR (`Display` 4101), no LiveKernelReport and no bugcheck/WHEA for this incident;
- `pnputil /restart-device` on the RTX 4090 released it (23,154 MiB → 2,014 MiB).

The server's own "the engine is frozen" line did not fire either, because it needs the engine to use no CPU, and this one was burning a core.

**C. The same machine produced three #29 stalls with stall dumps (context, not the bug above).**

13/11 15:44 (0.1.58), 14/11 23:30 (0.1.58) and 16/11 23:34 (0.1.60), all `reading the prompt (batched): waiting for the GPU (attention, router)`, 0 layers served, expert pool idle, host idle 63–79 s. `VRAM free when this request began` was 194 MiB (flagged `LOW`, #2065) and 1,493 MiB, with `the prompt's GPU work (GPU 0) was still running` on 0.1.60; RAM available 2,840–4,104 MiB. Each wrote `strata-stall-<pid>.dmp` and exited -6 (WER: 0xC0000409, subcode 7, faulting module `strata.exe`). Those dumps are still on this machine.

So this is the same family, in its **start-time** variant: no dump, and the GPU stays unusable.

### Smallest steps or command that reproduce it

Not a deterministic recipe yet. What was seen:

1. `strata runtime start --profile <profile>` on a profile with lazy loading and `idle_unload_s: 150`, used by an agent for a couple of hours (prompts around 125,000 tokens, loads, 150 s idle unloads, reloads).
2. A reload after an idle unload stalls at `strata serve: the prompt path borrows ...`. From then on every new attempt stalls at the same line until the NVIDIA device is restarted.

State at the stalled start (from the log):

- args: `--expert-cache auto --prefill auto --spec 4 --kv q4_0 --kv-resident 32768 --max-context 204800 --vram-reserve-mib 1792 --vision --vram-elastic`
- `expert cache auto: 17.95 GiB free, 2048 MiB reserved (+0 MiB for the draft head, +256 MiB for what is allocated after the cache) -> 7234 slots`, then `only 1574 MiB free once the slots are written (reserve 2048 MiB); shrinking`, then `expert cache 9362 slots, 15.17 GiB of VRAM`
- `the expert arena needs 39.97 GiB of RAM and 30.78 GiB is available ... (9.21 GiB short, after waiting)`, so `experts via mmap` and a 22.64 GiB page-locked RAM complement
- host: 62.9 GiB RAM, 7.6 GiB free at the time; the stall follows ~30 load/unload cycles in one session

### Reproduction status

Reproduced locally

### Strata version

Terminal 0.2.14 · Control 0.1.61 · engine 0.1.60 (windows-x86_64-cuda-cuda13, build 09b603886925+cuda13.0)

### Graphics card, memory and system

RTX 4090 24,564 MiB · 62.9 GiB RAM · Windows 11 Pro 10.0.26300 · NVIDIA driver 621.01 (32.0.16.2101), CUDA UMD 13.5

### Model, size and settings

Qwen3.8-Flash-Next IQ3_XXS (native pack, 2 GGUF shards + PLE shard) · `--max-context 204800` · `--kv q4_0 --kv-resident 32768` · pictures on (`--vision`; the reader falls back to the CPU here: 1,366 MiB of VRAM free and the GPU reader asks for about 1,536) · `--vram-reserve-mib 1792` during the incident (2,048 afterwards) · `--vram-elastic` · one request at a time

### Logs

Server, at the moment the start is abandoned (paths shortened):

```
[strata] still starting (929 s) - please wait ...
[strata] the engine did not start (try 2 of 3); again in 15 s
[strata] starting the engine: reading the model's weights ...
[strata] still starting (25 s) - please wait ...
```

Engine, last line of each stalled attempt, and the line that a healthy start prints next:

```
strata serve: the prompt path borrows 1870 CUDA0 cache slots (3.07 GiB)
   -> (healthy) strata hc: CUDA0: the hyper-connection read runs as staged ... checked bit for bit ...
```

Engine stall report from 16/11 23:34 (same family, during a request):

```
strata serve: no progress for 60 s during a request (reading the prompt (batched): waiting for the GPU (attention, router) at layer 1 of the prompt chunk from token 125013)
  VRAM free when this request began: 1493 MiB
  the prompt's GPU work (GPU 0) was still running
  expert pool: 24 of 24 jobs claimed, 24 done; 7 of 7 workers parked, 7 sleeping
  memory: 31844 MiB resident, 67153 MiB committed, 2840 MiB RAM available
```

### Asks

1. Arm the #29 watchdog support (the stall report and the thread dump) during the prompt-path init, so a start that stalls there leaves evidence instead of spinning silently.
2. Give the GPU completion waits in the start a bound, or fail the start with a reason, instead of waiting forever.
3. Make a stalled engine releasable: after `engine_ready_s`, the supervisor should end the process or say that it is stuck, rather than leaving 23 GB allocated and NVML blocked until the GPU device is restarted.
4. Is `--vram-reserve-mib 2048` (or a smaller `--max-context`) the setting to avoid this on a 24 GB card with `--vision`, given that the log reports 1,572–1,751 MiB free with everything loaded, and 194 MiB in the 14/11 stall?

Glad to attach the three `strata-stall-*.dmp` files and the full engine/server logs on request.

*Disclosure: written with AI assistance from this PC's logs and Windows event records.*

Related on this site

Quick links to related pages.