OpenEuroLLM Releases Complex KDA

Researchers from OpenEuroLLM have released Complex KDA (CKDA), which enhances the expressiveness of Kimi Delta Attention (KDA). While existing linear RNNs were limited in expressiveness due to low-rank corrections, CKDA implements 2D rotations by combining a single delta rule transformation with channel-wise gates.

Technical Features

• Range Extension: By combining two existing extension methods, the gate range is set to [-1, 1] and the delta rule coefficient β to [0, 2].
• Efficiency Maintenance: The transition matrix is diagonal-plus-rank-one and non-expansive, thus preserving the stability and efficiency of KDA.
• Expressiveness Proof: It was proven that all orthogonal diagonal-plus-rank-one matrices are CKDA transition matrices, and it can track all finite groups isomorphic to subgroups of SO(3).

Performance and Results

• Language Modeling: Outperforms Transformers and other linear RNNs, showing results similar to the base KDA model.
• Length Extrapolation: Achieved the strongest length extrapolation performance among the tested KDA range settings on S3, S4, and periodic audio continuous tasks.
• Layer Efficiency: Requires one fewer layer for state tracking compared to other diagonal-plus-rank-one linear RNNs.