[CVPR 2026] An Upcycling Method for Creating Truly Meaningful Experts within the Mixture-of-Experts Architecture

The existing Sparse Upcycling approach initializes multiple Experts by copying the FFN of a Dense model, which causes all Experts to end up with similar parameters, leading to the Expert Symmetry problem where specialization is diminished. This becomes a major cause of reduced model efficiency.

To address this, LG AI Research proposed the Cluster-Aware Upcycling technique. This approach differentiates Experts by leveraging the semantic structure of data contained in the Activation Space of the Dense model.

The core process is as follows:

• Step 1: Cluster FFN Activations using Spherical K-Means to distinguish semantic regions within the data.
• Step 2: Use Data-Aware Truncated SVD to initialize each Expert so that it handles the principal components of a specific cluster.
• Step 3: Use the cluster's Centroid as the Router Weight, guiding data to be assigned to the appropriate Expert from the early stages of training.

Additionally, to prevent specialization from being undermined by tokens with uncertain Routing during training, an EESD loss function leveraging an Ensemble Teacher is introduced to support stable training.