Issues / #2212
#2212 sudden crash on RTX 3070 for long task
open · @HiroshiMatsumoto · 0 comentarios · En GitHub
Setup & installAMD / HIPNVIDIA / CUDAModels & quants
Descripción
Thank you Niko1221-san for creating such a wonderful tool !!! I've been using this for a week now. Untill yesterday, I was running small errands such as small coding and OCR and translation tasks. Yesterday, i made it make batch for translation task for about 2,000 documents, not one whole document at once per iteration but small chunk of text, as small as surrounding context is retained. Sample tests went smoothly all perfection, but when it came to running batch execution for all documents, it crashes for unknown reason. By unknown reason, I mean I looked up log files under /var/log and looked at nvidia-smi query for any clues, only thing I found was graphic board gets lower voltage on nvidia-smi before crash. And by crash, what actually happens is i access this machine through ssh and run the task under tmux session and when this problem occurs it kicks me out of ssh and tmux sesion is gets killed. It seems as if it went on reboot, but i can ssh back in asap as i get kicked out. I tried several times but it occurs at uncertain points. it ouccred in the variant time-window; 10 mins, some longer minuts, over 1hr. It does not occur on certain translation-target document since it can pick up where it left and continues. There was no OOM error or kernel panic or anything in the log for around the time this incident occured. my setup is the following. CPU: AMD Ryzen 9 3950X 16-Core Processor MEM: DDR4 128GB Graphic: NVIDIA GeForce RTX 3070 8GB Disk Space: ``` $ df -Th | grep -iE "^/dev" /dev/nvme0n1p2 ext4 916G 455G 414G 53% / /dev/nvme1n1p1 ext4 234G 772M 221G 1% /media/XXXXXX /dev/sda ext4 916G 577G 293G 67% /media/YYYYYY ``` ``` bash $ cat /etc/os-release PRETTY_NAME="Ubuntu 24.04.5 LTS" NAME="Ubuntu" VERSION_ID="24.04" VERSION="24.04.5 LTS (Noble Numbat)" ``` Strata is on `/dev/sda` . Tested on both `/dev/nvme1n1p1` and `/dev/nvme0n1p2` diskspaces but same crash occurs. my setup.sh command is the following ``` bash ./setup.sh --setup --host 0.0.0.0 --api-key "XXXXXXXX" --model IQ3_XXS --context 65536 --vision cpu --yes ``` Is there anything anyone who can think of what might be the cause of this issue ? here is a part of the nvidia-smi log around the time crash occured. ``` timestamp, temperature.gpu, power.draw [W], utilization.gpu [%], memory.used [MiB], clocks.current.sm [MHz], clocks_event_reasons.active 2026/10/13 01:31:31.783, 69, 144.96 W, 100 %, 7559 MiB, 2010 MHz, 0x0000000000000000 2026/10/13 01:31:36.784, 69, 157.08 W, 100 %, 7559 MiB, 2010 MHz, 0x0000000000000000 2026/10/13 01:31:41.785, 70, 154.16 W, 100 %, 7559 MiB, 2010 MHz, 0x0000000000000000 2026/10/13 01:31:46.785, 70, 155.40 W, 100 %, 7559 MiB, 2010 MHz, 0x0000000000000000 2026/10/13 01:31:51.786, 70, 160.50 W, 100 %, 7559 MiB, 2010 MHz, 0x0000000000000000 2026/10/13 01:31:56.787, 69, 145.12 W, 93 %, 7559 MiB, 2010 MHz, 0x0000000000000000 2026/10/13 01:32:01.789, 69, 145.24 W, 100 %, 7559 MiB, 2010 MHz, 0x0000000000000000 2026/10/13 01:32:06.790, 64, 74.88 W, 100 %, 7559 MiB, 2025 MHz, 0x0000000000000000 2026/10/13 01:32:11.792, 62, 73.98 W, 100 %, 7559 MiB, 2025 MHz, 0x0000000000000000 2026/10/13 01:32:16.793, 61, 73.22 W, 100 %, 7559 MiB, 2025 MHz, 0x0000000000000000 2026/10/13 01:32:21.794, 61, 72.68 W, 100 %, 7559 MiB, 2040 MHz, 0x0000000000000024 2026/10/13 01:32:26.796, 60, 72.37 W, 100 %, 7559 MiB, 2040 MHz, 0x0000000000000000 2026/10/13 01:32:31.797, 59, 71.56 W, 100 %, 7559 MiB, 2040 MHz, 0x0000000000000000 2026/10/13 01:32:36.798, 59, 70.84 W, 100 %, 7559 MiB, 2040 MHz, 0x0000000000000000 ``` Thank you in advance !!
En el sitio
Enlaces a install, modelos, releases.