问题 / #2992

#2992 KV cache q8_0 + q5_1

open · @adbensi · 0 评论 · 去 GitHub 看

Models & quants

说明

Hello! Can I compile the KV cache with support for q8_0 + q5_1, or will that cause issues with the strata structure?

With 64 GB + two 16 GB graphics cards, and using Q8_0/Q5_1 quantization, there’s a good chance you’ll be able to use a 250K context window + IQ3_S, or not?

本站相关内容

相关页面的快捷入口。