Stepped MoE Framework Combines Elastic Structures and Sparse Gating for Adaptive LLM Inference

A new unified framework called Stepped MoE combines elastic structures with sparsely gated architectures to create LLMs that adapt simultaneously to deployment constraints and task requirements. Unlike existing approaches that treat these dimensions independently, this model conditions on both context and target efficiency specifications to control the accuracy-efficiency trade-off at inference time.

Key Capabilities and Results

The approach enables fine-grained control over model capacity by activating task-relevant parameters within elastically-nested sub-networks. This allows a single model to span multiple capacity points while maintaining input-adaptive routing.

Experimental results demonstrate the following advantages:

• Flexible Capacity: The model supports using 1, 2, 3, or 4 billion parameters dynamically.
• Performance Gain: It achieves 2-5% higher accuracy on knowledge-intensive benchmarks compared to dense counterparts.
• Efficiency: The model matches the latency metrics of dense models and performs on par with static versions of itself.
• Resource Optimization: By sharing model parameters, the framework saves device disk space and allows serving flexibility based on available DRAM and compute resources.