HuggingFace Unveils 3B Model SmolLM3

SmolLM3 is a 3B parameter scale model trained on 11T tokens, delivering performance that surpasses Llama-3.2-3B and Qwen2.5-3B.

Key features are as follows:

• Dual-mode reasoning: An instruct model supporting think and no_think modes.
• Multilingual support: Supports 6 languages including English, French, Spanish, German, Italian, and Portuguese.
• Long context: Supports up to 128k context by leveraging NoPE and YaRN.

Technically, it optimizes KV cache size during inference through GQA (Grouped Query Attention), and introduces the NoPE method that removes RoPE from certain layers to enhance long-context performance. It also applies Intra-Document Masking to ensure training stability.

HuggingFace has released not only the model but also an engineering blueprint including the architecture, data mixture ratios, and training methodology, enabling developers to understand and reproduce the model-building process.