反馈 / #2259
#2259 Patch - add Swift 1.5 - IQ4_XS | Q4_K_L | Q5_K_L | Q6_K
open · @adolfotregosa · 0 评论 · 去 GitHub 看
说明
I wanted to try those models. I have 128 GB of ram, so I was curious how Q5_K_L and Q6_K would run. With a single 3090 Q6_K I get 3X - 5X t/s, not bad at all. There’s no harm in having them as options on Strata. Q6_K requires -DSTRATA_MMQ_KQUANTS=ON -DSTRATA_Q6K_EXPERTS=ON Q4_K_L and Q5_K_L it is not a requirement but benefits from having -DSTRATA_MMQ_KQUANTS=ON [swift-iq4_xs-q4_k_l-q5_k_l-q6_k.patch](https://github.com/user-attachments/files/33382066/swift-iq4_xs-q4_k_l-q5_k_l-q6_k.patch)
本站相关内容
相关页面的快捷入口。