Falcon-H1 Hybrid LLM Released

The Falcon-H1 series offers a total of 6 open-source model sizes ranging from 0.5B to 34B parameters, including both Base and Instruct versions.

The core of this model lies in its hybrid architecture, which combines the Transformer's Attention mechanism with a State Space Model (SSM, Mamba-2) in parallel. This secures strong generalization performance while increasing inference speed and reducing memory usage.

Key features are as follows.

• Overwhelming Efficiency: The 0.5B model achieves performance on par with existing 7B models, and each model is designed to match or exceed the performance of models at least 2x larger than its actual size.
• Long Context Support: Supports up to 256K context length, optimized for processing long documents and complex reasoning.
• Multilingual and STEM Specialization: Natively supports 18 languages, and through intensive training on high-quality STEM data, demonstrates outstanding capabilities in mathematics and science.
• Flexible Deployment: Offered in various sizes from 0.5B to 34B, enabling wide-ranging use from edge devices to large-scale servers.

All models have been released as open-weight models under the Apache 2.0 license.