TII Unveils Falcon 3 Open Model Family

Falcon 3, developed by Abu Dhabi's TII (Technology Innovation Institute), is a family of decoder-only large language models (LLMs) with under 10B parameters. This release focuses on strengthening science, math, and coding capabilities.

Key Model Lineup:

• Falcon3-1B/3B-Base: Small models that maximize efficiency through Knowledge Distillation.
• Falcon3-Mamba-7B-Base: A model leveraging the SSM (State Space Model) architecture to enhance reasoning and math capabilities.
• Falcon3-7B/10B-Base: Transformer-based models, with the 10B model in particular aiming for SOTA performance among models under 13B.

Key Technology and Features:

• Depth up-scaling: The 10B model was built by duplicating and expanding the layers of the 7B model, maximizing performance.
• Llama compatibility: All transformer-based models are compatible with the Llama architecture, allowing easy integration into existing AI ecosystems.
• Support for various formats: Various quantized and variant versions are provided, including Instruct, GGUF, GPTQ, AWQ, and 1.58-bit.

Performance Metrics:

• Falcon3-10B-Base demonstrated excellent mathematical reasoning ability on MATH-Lvl5 (22.9) and GSM8K (83.0).
• Falcon3-3B-Base shows efficiency that surpasses larger-scale models such as Llama-3.1-8B.