AI BriefingModernBERT Released: The Next-Generation BERT Replacement
Key point
ModernBERT, a next-generation BERT replacement model with an 8,192-token long context and improved performance, has been released.
Details
Answer.AI and LightOn have released ModernBERT, a next-generation encoder model that overcomes the limitations of the existing BERT. ModernBERT delivers performance improvements that surpass existing models in both speed and accuracy.
Key Features:
- 8,192-token support: Significantly extends beyond the 512-token limitation of existing encoder models, enabling long-context processing.
- Model sizes: Available as Base (149M) and Large (395M) models.
- Code training: Trained on a large amount of code, optimizing it for code search and IDE functionality.
- High efficiency: Supports Flash Attention 2 to maximize processing speed.
ModernBERT is designed as a slot-in replacement that can be immediately applied to existing BERT-based tasks such as RAG, recommendation systems, and classification.
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.
Related
COLING 2025: AI Framework Unveiled for Restoring the Endangered Nüshu Language Using LLMs