Issues / #2651
#2651 Setup --calibrate doesn't test full RAM+VRAM MoE offload even if they fit and weights are on HDD
open · @ggrabarz · 0 comentarios · En GitHub
Setup & installNVIDIA / CUDALinux
Descripción
Title basically, got a 128GB RAM machine with a 5080 16GB and after running `./setup.sh --calibrate` had 10 tps decode. Hadn't I known you've got ngrams stored and streamed on HDD I'd be very disappointed in this performance and abandon it. That would be a shame since this engine deserves a 🥇. I think `--calibrate` should make sure that it tests for full RAM offload and warn the user that the disk is the bottleneck. ```text # Proxmox LXC GPU Inference Specs Host: ASUS MB OS: Ubuntu 26.04 (Resolute Raccoon) x86_64 Kernel: Linux 7.0.2-6-pve CPU: Intel(R) Xeon(R) E5-2673 v4 (8) @ 3.60 GHz GPU: NVIDIA GeForce RTX 5080 [Discrete] Memory: 96.96 GiB / 120.00 GiB (81%) Swap: 99.55 MiB / 7.91 GiB (1%) [ LXC Virtualization & CPU ] Type: lxc Available CPU Flags: avx2 [ NVIDIA NVML Status ] 615.78.08, NVIDIA GeForce RTX 5080, 16303 MiB, 7030 MiB, 8901 MiB [ Storage Mounts & Capacity ] Filesystem Size Used Avail Use% Mounted on /dev/loop0 503G 18G 461G 4% / /dev/loop1 2.0T 252G 1.7T 14% /dev/hdd [ LXC Device Nodes Mapping ] crw-rw-rw- 1 root root 195, 254 Oct 15 10:42 /dev/nvidia-modeset crw-rw-rw- 1 root root 507, 0 Oct 15 10:40 /dev/nvidia-uvm crw-rw-rw- 1 root root 507, 1 Oct 15 10:40 /dev/nvidia-uvm-tools crw-rw-rw- 1 root root 195, 0 Oct 15 10:40 /dev/nvidia0 crw-rw-rw- 1 root root 195, 255 Oct 15 10:40 /dev/nvidiactl /dev/nvidia-caps: cr-------- 1 root root 236, 1 Oct 15 10:40 nvidia-cap1 cr--r--r-- 1 root root 236, 2 Oct 15 10:40 nvidia-cap2 ```
En el sitio
Enlaces a install, modelos, releases.