Key Points of MoE (Mixture of Experts)

Mixture of Experts (MoE) is an architecture for efficiently scaling up a model's parameter size, and it has recently drawn significant attention with the emergence of models like Mixtral 8x7B.

Key Components

• Sparse MoE Layer: Replaces the existing dense FFN layer with multiple Expert networks.
• Gate Network (Router): Responsible for routing each token to the appropriate expert, and is optimized together with the model during training.

Main Advantages

• Training Efficiency: With the same computing resources, models much larger in scale than dense models can be pretrained faster.
• Inference Speed: Inference is faster compared to a dense model with the same number of parameters.

Key Challenges and Considerations

• Memory Usage: Since all experts must be loaded into memory, high VRAM is required.
• Training Stability: Load Balancing and Router Z-loss techniques are important to prevent tokens from being concentrated on specific experts.
• Fine-tuning: Fine-tuning is trickier compared to dense models, but it is being improved through recent instruction-tuning research.