Issues / #2259

#2259 Patch - add Swift 1.5 - IQ4_XS | Q4_K_L | Q5_K_L | Q6_K

open · @adolfotregosa · 0 comentarios · En GitHub

Models & quants

Descripción

I wanted to try those models. I have 128 GB of ram, so I was curious how Q5_K_L and Q6_K would run. 
With a single 3090 Q6_K I get 3X - 5X t/s, not bad at all. 

There’s no harm in having them as options on Strata.

Q6_K requires  -DSTRATA_MMQ_KQUANTS=ON -DSTRATA_Q6K_EXPERTS=ON 
Q4_K_L and  Q5_K_L it is not a requirement but benefits from having -DSTRATA_MMQ_KQUANTS=ON

[swift-iq4_xs-q4_k_l-q5_k_l-q6_k.patch](https://github.com/user-attachments/files/33382066/swift-iq4_xs-q4_k_l-q5_k_l-q6_k.patch)

En el sitio

Enlaces a install, modelos, releases.