Issues / #3238
#3238 [Engine] Linux: "exit code -9" with no OOM when started from a PipeWire desktop without rtkit (RLIMIT_RTTIME hard limit 0)
open · @vthomasino · 0 comentarios · En GitHub
Setup & installServer & APINVIDIA / CUDAModels & quantsWindowsLinux
Descripción
### Area [Engine] (speed, kernels, models, the server and API) ### Desktop app, Terminal or source? Source (START-HERE.bat, setup.sh or a build) ### What happened The engine sometimes dies at load (while it builds the RAM copy) or during a request: ``` [strata] the engine stopped unexpectedly (exit code -9). The usual cause is running out of RAM: Linux then ends the biggest program (check: sudo dmesg | grep -i -E 'killed process|out of memory'); ... ``` It was never RAM. There was no OOM kill in dmesg, `oom_kill` in /proc/vmstat did not move, systemd-oomd logged nothing, and auditd rules on kill/tkill/pidfd_send_signal saw no process send the signal. Since #215 the RAM hint only shows when the OS killed the engine; here the kernel did kill it, but for a different reason. **Cause, as far as I traced it** 1. The engine inherits `RLIMIT_RTTIME` = **0 / 0** (soft / hard) from the shell. Every terminal on this desktop has it (`ulimit -R` prints 0, and `/proc/<pid>/limits` shows 0/0 for bash and pi). 2. It comes from PipeWire's client module (`libpipewire-module-rt`). With rtkit not installed, the desktop portal answers for it, and the module lowers the limit: `mod.rt: clamping rt.time.soft from 18446744073709551615 to 0 because of RTKit` (PipeWire 1.6.8, xdg-desktop-portal 1.22.1). The desktop shell is a PipeWire client, so everything started from it inherits 0/0. With `rtkit.enabled = false` and `rtportal.enabled = false` in a copy of client.conf, a client keeps `unlimited`; `rlimits.enabled = false` alone does not change it. 3. With a hard limit of 0, the kernel's `check_thread_timers()` (kernel/time/posix-cpu-timers.c) sends SIGKILL as soon as it checks a thread with an armed per-thread CPU timer: `rt.timeout` is 0 for a normal thread, and `0 >= 0` passes. The thread does not need to be real-time. A tracefs `signal:signal_generate` trace caught one engine death: `sig=9 errno=0 code=128 comm=strata`, sent from inside the process by `posix_cpu_timers_work <- task_work_run <- exit_to_user_mode_loop`, not by another process. 4. A 15-line test program reproduces it outside Strata (`rttime-thread-timer.c` in the gist below). It calls `timer_create(CLOCK_THREAD_CPUTIME_ID)` with `SIGEV_SIGNAL`, arms 100 ms and spins for 2 s. From this shell it is killed with signal 9 (exit 137); under `systemd-run --user` (unlimited) it exits 0. With `SIGEV_NONE` it survives. **Not found yet:** which code in the engine process arms the per-thread CPU timer. Strata's sources do not call `timer_create`. Among the engine's loaded libraries, only libc imports it. In one traced session (load plus a 15K-token prompt), no engine thread called `timer_create` or `timer_settime`, and the engine survived. The deaths are intermittent: 4 in about 24 hours with UD-Q6_K_XL (during a 155K-token prefill, after a short reply, at load, and on a 16K-token prompt). `ulimit -r` is 0 too, so no thread can switch to SCHED_FIFO, and 900 samples of every thread's scheduling class during a 16K prefill showed none real-time. I have a trace of timer_create/timer_settime with user stacks running, and will add the caller here when it shows up. **Workaround used here:** start the server as a systemd user service, which takes the user manager's limits (unlimited) instead of the shell's. No -9 since (about 4 hours of use and A/B runs so far). ``` systemd-run --user --collect --unit=strata -p LimitRTTIME=infinity -- .venv/bin/python serve/server.py ... ``` Installing rtkit should also work (it answers 200 ms instead of 0). I did not test that. **Suggestion:** on Linux, when the engine ends with -9, check `resource.getrlimit(resource.RLIMIT_RTTIME)` in `serve/server.py` before printing the RAM hint. If the hard limit is finite, say so and name the fix (rtkit, or start Strata outside the desktop session's limits). A warning at start-up in the same case would catch it before the first crash. When the limit is unlimited, neither changes anything. I can send this as a small PR if it is wanted. ### Smallest steps or command that reproduce it 1. On a PipeWire desktop without rtkit, open a terminal and check `ulimit -R` (0 here). 2. Build and run `rttime-thread-timer.c` from https://gist.github.com/vthomasino/aef6f8e2413733cbe149a91178e889d8 : it is killed with signal 9. Run it again with `systemd-run --user --wait`: it exits 0. 3. The engine itself dies only now and then; it took long UD-Q6_K_XL sessions to see it (4 times in about 24 hours). ### Reproduction status Reproduced locally ### Strata version Strata 0.2.0-beta · engine 0.1.45.3 (tag v0.2.0-beta, b73dbce3); the deaths were on 0.1.45, the limit and the workaround are the same on 0.2.0-beta ### Graphics card, memory and system RTX 5090 Laptop 24 GB (driver 610.57.04) · 62 GB RAM · Core Ultra 9 275HX · Linux 7.2.5 (Omarchy/Arch, Hyprland), PipeWire 1.6.8, no rtkit ### Model, size and settings unsloth UD-Q6_K_XL · 256K context · pictures on · `--resident-budget-gib 38` ### Logs Kernel stack of the kill (tracefs, signal:signal_generate with stacktrace): ``` strata-371087 [006] dN.1. 307833.524553: signal_generate: sig=9 errno=0 code=128 comm=strata pid=371087 grp=1 res=0 => trace_event_raw_event_signal_generate => __send_signal_locked => send_signal_locked => posix_cpu_timers_work => task_work_run => exit_to_user_mode_loop => do_syscall_64 => entry_SYSCALL_64_after_hwframe ```
En el sitio
Enlaces a install, modelos, releases.