#2657 Support: Nemotron 3.5 Super VL 120B A12B

open · @emircanerkul · 0 comentários · No GitHub

Setup & install AMD / HIP Models & quants

Descrição

Looking forward to see news regarding that. I believe we can squeeze that into 16gb vram + 16gb ram setup. I use AMD 6800XT.

27B fits 7GB with 70tps via mtp
120B should fit 32GB with (due to A12B/27) 157tps via mtp.

It is indeed really tight. But in theory it should work with today's technology (without offloading anything to ssd)

But lets see how well it keep it's intelligent when scaled down/quantised

No site

Install Strata
Model sizes & RAM
Release notes

Links install, modelos, releases.