llama.cpp Dynamic KV Cache Quantization Feature

A new feature has been proposed for llama.cpp to address the trade-offs between VRAM usage, context length, and multimodal (mmproj) capability during LLM inference.

Previously, changing the KV cache quantization level or toggling mmproj usage required unloading and reloading the entire model, which caused significant delay in the process. To solve this, a feature that dynamically re-quantizes the KV Cache without reloading the model has been implemented.

Key implementation details:

• Added a POST /requantize_kvcache HTTP endpoint
• Reads and deletes the current KV cache, then creates a new empty cache at the desired quantization level
• Converts the existing KV cache into the new quantization format and loads it immediately

With this feature, it becomes possible to quickly switch between a high-precision KV cache (f16) and a VRAM-efficient quantized cache depending on the situation, even during an inference session, making it much more flexible to secure context length or enable multimodal capability.