OCTOPUS: Optimizing Transformer KV Cache via Optimal Squared-Error Quantization and Octahedral Parameterization

In autoregressive inference handling long-context, KV Cache is a key factor that accounts for most of the memory bandwidth and footprint. Existing rotation-preprocessing codecs such as TurboQuant and PolarQuant have attempted efficient compression by combining structured random rotations with scalar quantization.

OCTOPUS advances this paradigm a step further through a method that jointly quantizes rotated coordinate triplets. It maps the direction of each triplet onto a square via octahedral parameterization, and applies Lloyd-Max quantization to the resulting two coordinates and the triplet norm.

The key features of OCTOPUS are as follows:

• Optimal bit allocation: It optimizes the per-triplet squared error, adjusting bit allocation non-uniformly according to the overall dimensionality.
• Data independence: It is a data-oblivious online method that operates deterministically given a seed value.
• Outstanding compression performance: It outperforms existing rotation codecs across text, video, and audio domains, with the gap widening further in low-bit compression settings.

In addition, a fused implementation based on Triton enables real-time reconstruction of keys without separately generating uncompressed keys. This allows for efficient inference without any extra bandwidth consumption or latency compared to conventional dequantization.