Issues / #2651

#2651 Setup --calibrate doesn't test full RAM+VRAM MoE offload even if they fit and weights are on HDD

open · @ggrabarz · 0 comentarios · En GitHub

Setup & installNVIDIA / CUDALinux

Descripción

Title basically, got a 128GB RAM machine with a 5080 16GB and after running `./setup.sh --calibrate` had 10 tps decode. 
Hadn't I known you've got ngrams stored and streamed on HDD I'd be very disappointed in this performance and abandon it. 
That would be a shame since this engine deserves  a 🥇.

I think `--calibrate` should make sure that it tests for full RAM offload and warn the user that the disk is the bottleneck.

```text
# Proxmox LXC GPU Inference Specs

Host: ASUS MB
OS: Ubuntu 26.04 (Resolute Raccoon) x86_64
Kernel: Linux 7.0.2-6-pve
CPU: Intel(R) Xeon(R) E5-2673 v4 (8) @ 3.60 GHz
GPU: NVIDIA GeForce RTX 5080 [Discrete]
Memory: 96.96 GiB / 120.00 GiB (81%)
Swap: 99.55 MiB / 7.91 GiB (1%)

[ LXC Virtualization & CPU ]
Type: lxc
Available CPU Flags: avx2

[ NVIDIA NVML Status ]
615.78.08, NVIDIA GeForce RTX 5080, 16303 MiB, 7030 MiB, 8901 MiB

[ Storage Mounts & Capacity ]
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop0      503G   18G  461G   4% /
/dev/loop1      2.0T  252G  1.7T  14% /dev/hdd

[ LXC Device Nodes Mapping ]
crw-rw-rw- 1 root root 195, 254 Oct 15 10:42 /dev/nvidia-modeset
crw-rw-rw- 1 root root 507,   0 Oct 15 10:40 /dev/nvidia-uvm
crw-rw-rw- 1 root root 507,   1 Oct 15 10:40 /dev/nvidia-uvm-tools
crw-rw-rw- 1 root root 195,   0 Oct 15 10:40 /dev/nvidia0
crw-rw-rw- 1 root root 195, 255 Oct 15 10:40 /dev/nvidiactl

/dev/nvidia-caps:
cr-------- 1 root root 236, 1 Oct 15 10:40 nvidia-cap1
cr--r--r-- 1 root root 236, 2 Oct 15 10:40 nvidia-cap2
```

En el sitio

Enlaces a install, modelos, releases.