4 Key Techniques for Transformer Efficiency

This summary organizes key research addressing the quadratic increase in memory and time complexity of Transformer models as sequence length grows.

The main approaches are classified into the following four categories:

• Custom attention patterns: Achieves linear scaling by combining window-based or global attention, as in Longformer.
• Recurrence: Uses recurrent structures to compress and pass along information, as in Compressive Transformer.
• Low-rank approximations: Improves efficiency by approximating the attention matrix with a low-rank representation, as in Linformer.
• Kernel approximations: Approximates the attention mechanism using kernel functions, as in Performer.

These techniques enable efficient training and inference on longer sequences, going beyond the conventional 512 or 1024 token limits.