#2734 Hope add support for the LMCache framework

说明

First of all, thank you for creating such an excellent inference framework. It helps us get much more value out of the many 24 GB A10 GPUs we have.

I’ve deployed IQ3_S on a single A10 and am getting around 60 tokens per second. Our team has many users, so we’re seeing queues build up frequently.

I’d like to run four instances on the four A10 GPUs in the same server. However, since KV cache can’t currently be shared, I’d like to request support for LMCache so that multiple instances on the same server can share their KV cache.