What Is Mixture of Experts (MoE) — Why DeepSeek Runs Cheaply Despite Having 1.6 Trillion Parameters

Mixture of Experts (MoE) consists of multiple expert submodels and a router (gating network) that selects among them.

• Unlike a dense model, which uses all parameters for every token, MoE has the router activate only the few experts needed.
• As a result, even though the total parameter count is very large, the actual computation stays much smaller.

Based on DeepSeek V4-Pro, out of the total 1.6 trillion parameters, only 49 billion (about 3%) are activated per token.

• Thanks to this structure, the model holds knowledge at a 1.6 trillion scale while inference cost stays close to that of a 49 billion-scale model.
• MoE is generally described as improving price-to-performance by 3 to 5 times compared to a dense model of the same scale.

However, there are also constraints.

• All expert parameters must be loaded into memory, resulting in large VRAM requirements.
• The imbalance problem, where load concentrates on certain experts during training, must be separately managed.

In conclusion, as major frontier models adopt MoE, the explanation notes that the standard for AI model competition is shifting from simple size to price-to-performance.