Direct Performance Comparison of MoE vs Dense Models

This study directly compares and analyzes the MoE (Mixture-of-Experts) architecture and the conventional Dense architecture under the same compute budget environment.

The main comparison metrics are as follows:

• Performance and accuracy: Comparison of model prediction performance relative to parameter count
• Computational efficiency: Analysis of performance differences under the same FLOPs basis
• Inference efficiency: Measurement of latency and throughput by architecture

This study presents a threshold at which MoE models can maintain high performance with fewer active parameters compared to Dense models, and demonstrates the trade-offs in hardware resource utilization with objective data.