AI BriefingSSOG Attention Released to Reduce SDPA Complexity
Key point
A new attention mechanism called SSOG has been released, improving SDPA's $O(N^2)$ complexity to $O(N\sqrt{N})$.
Details
SSOG (Sum Of Separable Gaussians) is a new attention mechanism proposed to address the $O(N^2 \cdot d)$ complexity issue of existing SDPA (Scaled Dot-Product Attention).
This approach learns a small number of Gaussian atoms for each head and geometrically adjusts them based on query tokens. By leveraging the fact that Gaussian atoms can be factorized into a separable sum, it reduces computational complexity to $O(N \cdot \sqrt{N} \cdot d)$.
Key Features and Performance:
- Dataset Performance: It outperforms SDPA on smaller datasets like CIFAR-100, and shows comparable performance with faster convergence on larger datasets like ImageNet-1k (IN1k).
- Efficiency: It exhibits lower memory usage and faster processing speeds as the model scale increases.
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.