问题 / #3370

#3370 [Engine] serve: a temp cleaner that removes the idle vision encoder's work dir makes every later start fail ([WinError 267])

open · @smallmj · 2 评论 · 去 GitHub 看

Server & APINVIDIA / CUDAModels & quantsWindows

说明

### Area

[Engine] (the server and API)

### Desktop app, Terminal or source?

Source (the repo's `serve/server.py`, started by our own launcher/router)

### What happened

With `idle_unload_s` set, after the model had stayed unloaded for a while (overnight), the first request that loaded it again failed with a 503 whose message was:

```
the vision encoder failed to start: [WinError 267] The directory name is invalid. ; the next request restarts it
```

Every client retry failed the same way, seconds apart, for as long as the server process lived — only restarting the whole server healed it. The console showed a tight loop of full load attempts (the `before_load` hook ran, the free-VRAM check passed, then the vision start failed immediately; the engine was never reached):

```
[strata] model unloaded after 1800 s idle; the next request loads it again
[router] unload <other model>: 200 {"status": "not loaded"}
[strata] starting the vision encoder ...
[router] unload <other model>: 200 {"status": "not loaded"}
[strata] starting the vision encoder ...
... (x17, seconds apart, until a manual restart)
```

The engine's log had no trace of any of these attempts — no `engine started` lines (those are written before the spawn), no encoder stderr: the encoder process was never even spawned. Expected: the vision encoder starts again like it does after any other unload.

### Root cause

`Vision.work_dir()` runs once, at the server's start (`Vision.__init__`), and creates `%TEMP%\strata-vision-*`. The spawn uses `cwd=self.dir` and nothing ever checks that the directory still exists (`_start_reader()` at current HEAD; the single spawn path in earlier versions). `restart()` reuses the stale path.

While the model is unloaded, that directory sits **empty and unheld** in the temp directory — exactly what Windows' Storage Sense ("Delete temporary files that my apps aren't using", enabled on the affected machine) removes during a long idle window. The next request's `popen(cwd=<removed dir>)` raises `OSError [WinError 267]`, which `ensure_loaded` wraps as `EngineDied("the vision encoder failed to start: ...")` → 503 → the client retries → the identical failure, forever. `#914` made `shutdown()` the only remover of the dir, but nothing defends it against external removal.

Ruled out on the affected machine while diagnosing:

- **not VRAM**: with the engine resident and only ~850 MiB of VRAM free, a manually spawned `strata-vision.exe --gpu` with the config's real arguments started fine (mmproj load, warm-up, READY);
- **not a crash**: the Windows Event Log has no strata crash entries, and the encoder leaves no stderr at all — it never ran;
- the engine itself is never started during the loop.

### Smallest steps that reproduce it

1. Start the server `--lazy` with a vision config.
2. Delete the `strata-vision-*` directory in the temp directory.
3. Send any completion request (the encoder starts with the model load, not with a picture) → the 503 loop above, until the server is restarted.

### Reproduction status

Reproduced locally (twice on the affected machine, on different days; the second time with the client error text and the console loop captured).

### Strata version

engine 0.1.45 · serve/server.py from this repo (the deployed checkout was unmodified; the bug is confirmed present at current main by reading the code, e88ee39d)

### Graphics card, memory and system

RTX 3090 24 GB · 64 GB RAM · Windows 11 Pro

### Model, size and settings

Qwen3.8-Flash-Next IQ3_S · 512K context (yarn ×2, k8v4 KV) · pictures on (`--vision`, GPU encoder)

### Fix

In `Vision._start()` (the wrapper at HEAD, so the CPU-fallback path is covered too), recreate the dir when it is gone and drop the cache — the encoded-image files lived in that dir, and the cache-hit branch of `encode()` returns paths without checking they exist:

```python
if not self.dir.exists():                       # a temp cleaner (Windows' Storage Sense, systemd-tmpfiles)
    self.dir = self.work_dir()                  # can remove the idle encoder's dir while the model is
    self.cache.clear()                          # unloaded: every start would then fail on the cwd, and the
                                                # cache's files went with the dir
```

PR #3372 carries the fix and a regression test (`serve/test_vision_dir.py`).

本站相关内容

相关页面的快捷入口。