Exploring DeltaNet-family Linear Attention Variants

This mathematically analyzes the complex structures of recently spotlighted Linear Attention variant models and clearly explains what each model aims to achieve.

The main models analyzed are as follows:

• Kimi Delta Attention (KDA): Covers the state update equations and the process of mapping from Key space to Value space.
• DeltaNet family: Explains the state update mechanism and the correction process via error, using Bra-ket notation.

This piece breaks down the State Update, Error Calculation, and final Output generation process of each model into equations, providing technical insight into the logical flow that lies behind linear attention variants that may otherwise seem complex.