AI BriefingMoBA: MoE-based Block Attention Unveiled
Key point
MoBA, a technology that applies the MoE principle to attention to maximize the efficiency of long-context processing, has been announced.
Details
MoBA (Mixture of Block Attention), announced by Moonshot AI and Tsinghua University researchers, is a new technique designed to solve the attention computation cost problem that arises during long-context processing.
Unlike existing sliding window or linear attention methods, MoBA divides the entire context into multiple Blocks and is designed so that each query token selects the most relevant block on its own. In this process, it leverages the top-k gating mechanism of Mixture of Experts (MoE) to achieve computational sparsity.
Key Results and Features:
- High Efficiency: It recorded up to 6.5x faster speed than full attention at 1 million token prefill, and up to 16x faster at 10 million tokens.
- Maintained Performance: While dramatically reducing computation, it maintains nearly the same performance as full attention across various benchmarks.
- Real-world Deployment: It has already been applied to Moonshot AI's Kimi model, where it is used to handle long-context requests.
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.
Related
Trajectory Introduces Density-Aware Training to Optimize AI Cost-Per-TaskFireworks Releases Ember-1: Matches Kimi K3's Performance with 40% Fewer Tokens