Qwen1.5-MoE: Achieving 7B Model Performance with Only 1/3 of the Activated Parameters

Qwen1.5-MoE-A2.7B is a small MoE model that delivers performance comparable to state-of-the-art 7B models such as Mistral 7B and Qwen1.5-7B, using only 2.7 billion activated parameters.

Compared to the existing Qwen1.5-7B, its number of non-embedding parameters is only about 1/3 (2 billion). This reduces training costs by 75% and improves inference speed by 1.74x, maximizing resource efficiency.

The model's core architecture includes the following three innovations:

• Fine-grained experts: A single FFN is split into multiple segments, resulting in a total of 64 experts—8 times more than the existing 8-expert structure.
• Upcycling: Instead of training the model from scratch, the existing Qwen-1.8B is recycled for initialization, improving convergence speed and performance.
• Routing mechanism: Combines 4 shared experts that are always activated with a routing experts structure that selects 4 out of 60 experts, ensuring flexibility.

On major benchmarks such as MMLU, GSM8K, and HumanEval, it has demonstrated excellent performance competitive with top 7B models like Mistral-7B and Gemma-7B.