MiniMax Unveils MSA for Ultra-Long-Context Processing

MiniMax has unveiled MiniMax Sparse Attention (MSA), a new attention mechanism for Ultra-long-context processing. MSA is a block-wise sparse attention technology based on Grouped Query Attention (GQA).

Core Technical Features:

• Index Branch: Scores KV blocks to independently select the optimal Top-k subset for each GQA group.
• Main Branch: Performs precise block sparse attention only on the selected blocks.
• GPU Optimization: Co-designed dedicated kernels that enhance Tensor Core utilization through exp-free Top-k selection and KV-outer sparse attention.

Performance Metrics (based on the 109B model):

• Maintains performance on par with GQA at 1M token context while reducing attention computation per token by 28.4x.
• When MSA is applied on H800 GPU environments, it shows wall-clock performance improvements of 14.2x for Prefill speed and 7.6x for Decoding speed.

The related code and models have been released via GitHub and Hugging Face.