Huawei Unveils 505B-Parameter MoE Model

Huawei has unveiled openPangu-2.0-Pro, a new MoE (Mixture-of-Experts) model trained on Ascend accelerators. The model has a total of 505B parameters, with a structure in which only 18B parameters are activated during inference.

Key features are as follows:

• Context length: Supports 512k
• Training data: A total of 34T tokens used
• Training method: Unified SFT (Supervised Fine-Tuning) combining Slow and Fast thinking capabilities, multi-expert RL (Reinforcement Learning) training, and On-policy distillation techniques combining multiple RL experts

Detailed information about the model can be found in the openPangu-2.0 technical report.