Llama 4 Maverick and Scout Unveiled

Meta's next-generation large language models, Llama 4 Maverick (~400B) and Llama 4 Scout (~109B), have been released on the Hugging Face Hub. Both models adopt a Mixture of Experts (MoE) architecture, and both share the same 17B active parameters.

Key Features and Architecture:

• Native Multimodal: Uses an early fusion approach capable of processing both text and image inputs.
• Massive Context Support: Supports 256K during pretraining, and for the Instruct models, Maverick offers a context length of up to 1M, while Scout offers up to 10M.
• NoPE (No RoPE) Layers: Instead of conventional RoPE, a NoPE layer is used every 4 layers, optimized for long-context processing. Chunked attention is applied to the remaining layers where RoPE is used.

Deployment and Ecosystem Integration:

• Hugging Face Integration: Immediately compatible with transformers (v4.51.0) and TGI, making model loading, inference, and fine-tuning easy.
• Quantization Support: Scout supports 4-bit/8-bit quantization for single-GPU deployment, while Maverick supports the FP8 format.
• Xet Storage: A Xet storage backend has been introduced to improve model upload and download efficiency, enhancing data deduplication and speed.