Meta Expert Architecture Matches Vanilla MoE Performance Using 70-80% of Experts

·2026.10.23 14:51

Key point

Cross-layer shared Meta Experts reduce parameter redundancy and improve downstream task accuracy compared to hierarchical Local Experts.

Meta Expert architecture shares experts across layers to eliminate inter-layer redundancy, achieving lower cross-entropy loss and higher downstream task accuracy than Local Experts.
With activated quantity and expert granularity fixed, the model matches Vanilla MoE performance while using only 70-80% of the experts.
Theoretical analysis indicates Meta Experts provide more flexible residual pathways for hidden states, improving information retrieval efficiency.
Empirical comparison confirms Meta Experts exhibit lower parameter redundancy compared to hierarchically arranged Local Experts.

This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.

Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.