AI BriefingNyströmformer: Efficient Approximation
Key point
We introduce Nyströmformer, which uses the Nyström method to reduce the complexity of Transformer's self-attention from $O(n^2)$ to $O(n)$.
Details
The self-attention mechanism at the core of Transformer has $O(n^2)$ time and memory complexity proportional to the square of the input sequence length ($n$), leading to enormous costs when processing long sequences.
Nyströmformer approximates this to $O(n)$ complexity by leveraging the Nyström method, a matrix approximation technique. This method uses the principle of approximating a matrix by sampling some rows and columns instead of computing the entire matrix.
The key features are as follows:
- Landmark Sampling: To address the difficulty of directly applying the Nyström method to the softmax matrix, landmarks (Nyström points) are sampled from the Query and Key.
- Segment Means: Landmarks are generated by dividing the sequence into multiple segments and taking the mean of each segment.
- Performance Maintained: With only a small number of landmarks, it shows performance comparable to standard self-attention and existing efficient attention mechanisms on long sequences.
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.
Related
The Normalization of AI Translation: Principles of LLM-Based Translation and Structural Differences from NMT