Speculative KV Coding: Losslessly Compressing the KV Cache by Up to ~4x

As LLM context lengths grow, the size of the KV cache is emerging as a major bottleneck for memory usage and data movement. Existing approaches such as TurboQuant use lossy compression by reducing bit width, but this has the limitation that the resulting degradation in model performance is difficult to predict in advance.

Speculative KV coding presents a new methodology that uses a predictor model to achieve lossless compression of the KV cache.

Key mechanism:

• Using a Predictor Model: A model that is lighter and faster than the target model predicts the target model's cache in advance for the same prompt.
• Arithmetic Coding: Depending on how well the predictor model matches the actual cache, the actual cache values are encoded only as the difference from the predicted values.
• Similarity to Speculative Decoding: The closer the predictor model's output is to the actual result, the higher the compression efficiency—applying the principle of Speculative Decoding to cache compression.

Through this technique, the KV cache can be compressed by up to about 4x, dramatically improving memory efficiency without any degradation in model quality.