TII Releases Falcon Mamba 7B Model

TII (Technology Innovation Institute) has released the Falcon Mamba 7B model, designed to overcome the limitations of the attention mechanism found in existing Transformers. This model is based on a pure Mamba (State Space Model, SSM) architecture.

Key Technical Features:

• Sequence Scalability and Efficiency: By solving the computation/memory cost problems of attention, it maintains constant token generation time regardless of sequence length, allowing it to process sequences of arbitrary length.
• Memory Optimization: It operates without memory increasing with sequence length, making it possible to run on a single A10 (24GB) GPU.
• Training Stability: An additional RMS normalization layer was introduced into the Mamba architecture to ensure stability during large-scale training.

Training and Performance Results:

• Dataset: It was trained on approximately 5,500GT tokens, including RefinedWeb and high-quality technical/code data.
• Benchmark Performance: On key metrics such as IFEval, BBH, and MMLU-PRO, it recorded performance on par with or superior to existing Transformer-based models and hybrid models such as Llama 3 and Mistral, demonstrating strong competitiveness as a pure SSM model.