MoE Model Maintains Performance Even After Removing 28% of Routed Experts

Testing Routing Tail redundancy during inference in a Mixture-of-Experts (MoE) model revealed that removing low-weight experts causes almost no performance degradation.

Key experiment details:

• Target model: 35B MoE (top-k=8, total experts=256)
• Methodology: Instead of reducing the fixed k value, applied a per-token thresholding approach that skips experts whose router-assigned weight falls below a certain threshold (1/k)
• Results: Even after removing about 28% of routed experts, GSM8K benchmark performance remained the same as before (12/15) or showed a slight improvement (13/15)

Key implications:

• MoE model routers concentrate weight on specific experts, and lower-weight experts have redundancy, contributing very little to the output
• This research suggests the possibility of dynamically skipping low-weight experts during inference to improve computational efficiency
• However, the sample size was small (n=15), so broader benchmarks and large-scale validation are needed