#2992 KV cache q8_0 + q5_1

Beschreibung

Hello! Can I compile the KV cache with support for q8_0 + q5_1, or will that cause issues with the strata structure?

With 64 GB + two 16 GB graphics cards, and using Q8_0/Q5_1 quantization, there’s a good chance you’ll be able to use a 250K context window + IQ3_S, or not?