Issues / #2212

#2212 sudden crash on RTX 3070 for long task

open · @HiroshiMatsumoto · 0 commentaires · Sur GitHub

Setup & installAMD / HIPNVIDIA / CUDAModels & quants

Description

Thank you Niko1221-san for creating such a wonderful tool !!!

I've been using this for a week now.
Untill yesterday, I was running small errands such as small coding and OCR and translation tasks.
Yesterday, i made it make batch for translation task for about 2,000 documents, not one whole document at once per iteration but small chunk of text, as small as surrounding context is retained.
Sample tests went smoothly all perfection, but when it came to running batch execution for all documents, it crashes for unknown reason.  
By unknown reason, I mean I looked up log files under /var/log and looked at nvidia-smi query for any clues, only thing I found was graphic board gets lower voltage on nvidia-smi before crash.
And by crash, what actually happens is i access this machine through ssh and run the task under tmux session and when this problem occurs it kicks me out of ssh and tmux sesion is gets killed. 
It seems as if it went on reboot, but i can ssh back in asap as i get kicked out.
I tried several times but it occurs at uncertain points.

it ouccred in the variant time-window; 10 mins, some longer minuts, over 1hr.
It does not occur on certain translation-target document since it can pick up where it left and continues.

There was no OOM error or kernel panic or anything in the log for around the time this incident occured.

my setup is the following.
CPU: AMD Ryzen 9 3950X 16-Core Processor
MEM: DDR4 128GB
Graphic: NVIDIA GeForce RTX 3070 8GB
Disk Space: 
```
$ df -Th | grep -iE "^/dev"
/dev/nvme0n1p2 ext4      916G  455G  414G  53% /
/dev/nvme1n1p1 ext4      234G  772M  221G   1% /media/XXXXXX
/dev/sda       ext4      916G  577G  293G  67% /media/YYYYYY
```

``` bash
$ cat /etc/os-release
PRETTY_NAME="Ubuntu 24.04.5 LTS"
NAME="Ubuntu"
VERSION_ID="24.04"
VERSION="24.04.5 LTS (Noble Numbat)"
```

Strata is on `/dev/sda` .
Tested on both `/dev/nvme1n1p1` and `/dev/nvme0n1p2` diskspaces but same crash occurs. 

my setup.sh command is the following
``` bash
./setup.sh  --setup --host 0.0.0.0 --api-key "XXXXXXXX" --model IQ3_XXS --context 65536 --vision cpu --yes
```

Is there anything anyone who can think of what might be the cause of this issue ?

here is a part of the nvidia-smi log around the time crash occured.

```
timestamp, temperature.gpu, power.draw [W], utilization.gpu [%], memory.used [MiB], clocks.current.sm [MHz], clocks_event_reasons.active
2026/10/13 01:31:31.783, 69, 144.96 W, 100 %, 7559 MiB, 2010 MHz, 0x0000000000000000
2026/10/13 01:31:36.784, 69, 157.08 W, 100 %, 7559 MiB, 2010 MHz, 0x0000000000000000
2026/10/13 01:31:41.785, 70, 154.16 W, 100 %, 7559 MiB, 2010 MHz, 0x0000000000000000
2026/10/13 01:31:46.785, 70, 155.40 W, 100 %, 7559 MiB, 2010 MHz, 0x0000000000000000
2026/10/13 01:31:51.786, 70, 160.50 W, 100 %, 7559 MiB, 2010 MHz, 0x0000000000000000
2026/10/13 01:31:56.787, 69, 145.12 W, 93 %, 7559 MiB, 2010 MHz, 0x0000000000000000
2026/10/13 01:32:01.789, 69, 145.24 W, 100 %, 7559 MiB, 2010 MHz, 0x0000000000000000
2026/10/13 01:32:06.790, 64, 74.88 W, 100 %, 7559 MiB, 2025 MHz, 0x0000000000000000
2026/10/13 01:32:11.792, 62, 73.98 W, 100 %, 7559 MiB, 2025 MHz, 0x0000000000000000
2026/10/13 01:32:16.793, 61, 73.22 W, 100 %, 7559 MiB, 2025 MHz, 0x0000000000000000
2026/10/13 01:32:21.794, 61, 72.68 W, 100 %, 7559 MiB, 2040 MHz, 0x0000000000000024
2026/10/13 01:32:26.796, 60, 72.37 W, 100 %, 7559 MiB, 2040 MHz, 0x0000000000000000
2026/10/13 01:32:31.797, 59, 71.56 W, 100 %, 7559 MiB, 2040 MHz, 0x0000000000000000
2026/10/13 01:32:36.798, 59, 70.84 W, 100 %, 7559 MiB, 2040 MHz, 0x0000000000000000
```

Thank you in advance !!

Sur le site

Liens install, modèles, releases.