AI BriefingEttin: SoTA Encoder-Decoder Models
Key point
The Ettin Suite has been released, a family of SoTA-level encoder and decoder models trained with the same data and training recipe.
Details
Ettin applies ModernBERT's training recipe to decoder-only models, making it the first SoTA model family to train encoders (MLM) and decoders (CLM) under identical conditions (2T tokens, same architecture and recipe). This enables a true 'apples-to-apples' comparison between architectures.
Key features include:
- Diverse model scales: Available in 6 sizes ranging from 17M to 1B parameters, covering everything from on-device environments to high-performance tasks.
- 3-stage training process: Optimized through 1.7T tokens of pre-training, 250B tokens of training for 8K context extension, and a 100B token decay phase based on high-quality data.
- Reproducible data: Unlike ModernBERT, all training data is publicly available, making research and reproduction easier.
Notably, the Ettin decoder model outperformed Llama 3.2 1B and SmolLM2, demonstrating state-of-the-art performance among open-data-based models.
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.