#4967 [Engine] serve: warn when systemd-oomd is running, and name it when the engine is ended by signal 9

open · @cakescats · 0 commentaires · Sur GitHub

Setup & installServer & APIAMD / HIPNVIDIA / CUDAWindowsLinux

Description

## Problem and result
systemd-oomd is on by default on Fedora and Ubuntu 22.04+. When memory pressure stays high it ends whole programs with SIGKILL, before the kernel's OOM killer and without a dmesg line. The engine fills most of the RAM with the model, so oomd can end the engine, or the browser and other apps beside it. Strata said nothing about this: the death note on signal 9 sent the user to `dmesg`, which shows nothing after an oomd kill.

Now:
- At the start of `serve` (engine `strata`), a running systemd-oomd is said in one `[strata] WARNING:` line, the same way the RLIMIT_RTTIME note (#3238) is.
- When the engine is ended by signal 9 and oomd is running, the death note names oomd, its journal (`journalctl -u systemd-oomd`) and the fix (close other programs, a smaller model, or `sudo systemctl mask --now systemd-oomd`), instead of the dmesg hint.

Nothing is changed or refused. On Windows, macOS and a Linux without oomd the output is the same as before.

## Related discussion
none (a warning only, no change to a default)

## Approach
`oomd_running()` reads `/proc/*/comm`, with no `systemctl` call, and returns False when `/proc` cannot be read. `oomd_note()` returns the sentence, or `""` outside Linux or without oomd. In the death note the RTTIME check stays first; the oomd note comes before the generic RAM hint, only for rc == -9.

## Validation
**Passed:**
- `python3 -m unittest serve.test_server` on Ubuntu 24.04.5 x86_64 (Python 3.12.3, a QEMU VM), branch head: 361 tests, OK.
- Same VM, without systemd-oomd: `oomd_note()` is `""`.
- Same VM, `apt install systemd-oomd` ([email protected] has ManagedOOMMemoryPressure=kill): `python3 serve/server.py --engine strata` prints the warning as its first line; `--engine mock` does not.
- Same VM, a real oomd kill: a stand-in engine (a Python process touching 700 MB in a user scope with MemoryHigh=150M, 1 GB swap) started with `subprocess.Popen` as the server does. The oomd pressure limit for [email protected] was lowered to 10% with `systemctl set-property --runtime` so the kill came in minutes. The journal says `Killed .../strata-fake-engine.scope due to memory pressure ... 16.59% > 10.00% for > 20s`, the process returned rc=-9, `StrataEngine.death_note()` named systemd-oomd and its journal, and `dmesg` had no "killed process" line.

**Failed:**
- On macOS 26 arm64 with Python 3.9, 3 tests in `serve.test_server` error (AnswerBeforeTheBody socket timeout, two VisionCacheEviction tests). They error the same way without this change; on the Linux VM all pass.

**Not run:**
- A real model on a real GPU under oomd: the stand-in process stands for the engine. No C++/CUDA/HIP/SYCL file is touched.
- Fedora and other distributions (only Ubuntu 24.04 tested).

## Limits
- The check says oomd is running, not that it watches the cgroup the server is in. On Fedora and Ubuntu it watches every user session ([email protected]), but a server started from a system unit outside that may never be its target and still sees the warning.
- On a signal 9 with oomd running, a kernel OOM kill is not told apart from an oomd kill; the note points to oomd's journal, where an oomd kill is logged.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Sur le site

Liens install, modèles, releases.