TII Releases Open-Source LLM 'Falcon'

Falcon, a state-of-the-art family of language models developed by Abu Dhabi's TII (Technology Innovation Institute), has been released under the Apache 2.0 license.

The main model lineup is as follows:

• Falcon-40B: A 'truly open' model with performance comparable to closed-source models, ranking at the top of the Open LLM Leaderboard at the time of its release.
• Falcon-7B: An efficient model capable of inference and fine-tuning even on consumer-grade hardware.
• Falcon-180B: A large-scale model capable of competing with PaLM-2.

A key differentiator of the Falcon models is the use of the RefinedWeb dataset. It leverages high-quality data that has undergone deduplication and rigorous filtering from large-scale web data, improving training efficiency.

In addition, Multi-Query Attention (MQA) technology has been applied to significantly improve inference-time scalability. This is a method in which all attention heads share the Key and Value embeddings, optimizing inference performance.

Users can easily perform inference, quantization, and PEFT-based fine-tuning of Falcon models through the Hugging Face ecosystem.