Issues / #3238

#3238 [Engine] Linux: "exit code -9" with no OOM when started from a PipeWire desktop without rtkit (RLIMIT_RTTIME hard limit 0)

open · @vthomasino · 0 comentários · No GitHub

Setup & installServer & APINVIDIA / CUDAModels & quantsWindowsLinux

Descrição

### Area

[Engine]  (speed, kernels, models, the server and API)

### Desktop app, Terminal or source?

Source (START-HERE.bat, setup.sh or a build)

### What happened

The engine sometimes dies at load (while it builds the RAM copy) or during a request:

```
[strata] the engine stopped unexpectedly (exit code -9). The usual cause is running out of RAM: Linux then ends the biggest program (check: sudo dmesg | grep -i -E 'killed process|out of memory'); ...
```

It was never RAM. There was no OOM kill in dmesg, `oom_kill` in /proc/vmstat did not move, systemd-oomd logged nothing, and auditd rules on kill/tkill/pidfd_send_signal saw no process send the signal. Since #215 the RAM hint only shows when the OS killed the engine; here the kernel did kill it, but for a different reason.

**Cause, as far as I traced it**

1. The engine inherits `RLIMIT_RTTIME` = **0 / 0** (soft / hard) from the shell. Every terminal on this desktop has it (`ulimit -R` prints 0, and `/proc/<pid>/limits` shows 0/0 for bash and pi).
2. It comes from PipeWire's client module (`libpipewire-module-rt`). With rtkit not installed, the desktop portal answers for it, and the module lowers the limit: `mod.rt: clamping rt.time.soft from 18446744073709551615 to 0 because of RTKit` (PipeWire 1.6.8, xdg-desktop-portal 1.22.1). The desktop shell is a PipeWire client, so everything started from it inherits 0/0. With `rtkit.enabled = false` and `rtportal.enabled = false` in a copy of client.conf, a client keeps `unlimited`; `rlimits.enabled = false` alone does not change it.
3. With a hard limit of 0, the kernel's `check_thread_timers()` (kernel/time/posix-cpu-timers.c) sends SIGKILL as soon as it checks a thread with an armed per-thread CPU timer: `rt.timeout` is 0 for a normal thread, and `0 >= 0` passes. The thread does not need to be real-time. A tracefs `signal:signal_generate` trace caught one engine death: `sig=9 errno=0 code=128 comm=strata`, sent from inside the process by `posix_cpu_timers_work <- task_work_run <- exit_to_user_mode_loop`, not by another process.
4. A 15-line test program reproduces it outside Strata (`rttime-thread-timer.c` in the gist below). It calls `timer_create(CLOCK_THREAD_CPUTIME_ID)` with `SIGEV_SIGNAL`, arms 100 ms and spins for 2 s. From this shell it is killed with signal 9 (exit 137); under `systemd-run --user` (unlimited) it exits 0. With `SIGEV_NONE` it survives.

**Not found yet:** which code in the engine process arms the per-thread CPU timer. Strata's sources do not call `timer_create`. Among the engine's loaded libraries, only libc imports it. In one traced session (load plus a 15K-token prompt), no engine thread called `timer_create` or `timer_settime`, and the engine survived. The deaths are intermittent: 4 in about 24 hours with UD-Q6_K_XL (during a 155K-token prefill, after a short reply, at load, and on a 16K-token prompt). `ulimit -r` is 0 too, so no thread can switch to SCHED_FIFO, and 900 samples of every thread's scheduling class during a 16K prefill showed none real-time. I have a trace of timer_create/timer_settime with user stacks running, and will add the caller here when it shows up.

**Workaround used here:** start the server as a systemd user service, which takes the user manager's limits (unlimited) instead of the shell's. No -9 since (about 4 hours of use and A/B runs so far).

```
systemd-run --user --collect --unit=strata -p LimitRTTIME=infinity -- .venv/bin/python serve/server.py ...
```

Installing rtkit should also work (it answers 200 ms instead of 0). I did not test that.

**Suggestion:** on Linux, when the engine ends with -9, check `resource.getrlimit(resource.RLIMIT_RTTIME)` in `serve/server.py` before printing the RAM hint. If the hard limit is finite, say so and name the fix (rtkit, or start Strata outside the desktop session's limits). A warning at start-up in the same case would catch it before the first crash. When the limit is unlimited, neither changes anything. I can send this as a small PR if it is wanted.

### Smallest steps or command that reproduce it

1. On a PipeWire desktop without rtkit, open a terminal and check `ulimit -R` (0 here).
2. Build and run `rttime-thread-timer.c` from https://gist.github.com/vthomasino/aef6f8e2413733cbe149a91178e889d8 : it is killed with signal 9. Run it again with `systemd-run --user --wait`: it exits 0.
3. The engine itself dies only now and then; it took long UD-Q6_K_XL sessions to see it (4 times in about 24 hours).

### Reproduction status

Reproduced locally

### Strata version

Strata 0.2.0-beta · engine 0.1.45.3 (tag v0.2.0-beta, b73dbce3); the deaths were on 0.1.45, the limit and the workaround are the same on 0.2.0-beta

### Graphics card, memory and system

RTX 5090 Laptop 24 GB (driver 610.57.04) · 62 GB RAM · Core Ultra 9 275HX · Linux 7.2.5 (Omarchy/Arch, Hyprland), PipeWire 1.6.8, no rtkit

### Model, size and settings

unsloth UD-Q6_K_XL · 256K context · pictures on · `--resident-budget-gib 38`

### Logs

Kernel stack of the kill (tracefs, signal:signal_generate with stacktrace):

```
strata-371087  [006] dN.1. 307833.524553: signal_generate: sig=9 errno=0 code=128 comm=strata pid=371087 grp=1 res=0
 => trace_event_raw_event_signal_generate
 => __send_signal_locked
 => send_signal_locked
 => posix_cpu_timers_work
 => task_work_run
 => exit_to_user_mode_loop
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
```

No site

Links install, modelos, releases.