Meta Releases MoE LLM for Mobile

Meta has released the MobileMoE series to push the boundaries of quality and efficiency for on-device LLMs. This model adopts a Mixture-of-Experts(MoE) architecture and offers an INT4 weight footprint of less than 3GB to fit mobile DRAM.

The released models come in three scales: S/M/L, each with 0.3B/0.5B/0.9B active parameters respectively. Notably, MobileMoE-L maximizes efficiency by activating only 922M out of 5.3B total parameters. Each scale is provided in three variants: Base (pre-trained and mid-trained), SFT (instruction-tuned), and QAT (quantization-aware training).

Key technical specifications are as follows:

• Active Parameters: 922M (for the L model)
• Total Parameters: 5.3B
• Layers: 32
• Routed Experts: 60 (4 activated per token)
• Shared Expert: 1 (always activated)
• Context Length: 8,192 tokens
• License: FAIR NC