Amazon releases HOPE for higher-order pruning of MoE language models

Amazon published HOPE (Higher-Order Pruning of Experts), a method to reduce memory overhead in Mixture-of-Experts (MoE) models by accounting for pairwise expert interactions during pruning.
HOPE achieves an average rank of 2.07 across five methods and 39% of conditions, significantly outperforming first-order baselines like REAP at high pruning ratios.
On SWE-Bench Pro with Qwen3.5-35B-A3B at 50% pruning, HOPE scores 42.3 compared to REAP's 36.2, preserving performance close to the unpruned model (41.8).
The approach formulates pruning as a Quadratic Program (QP) to minimize error, requiring only a single forward pass for calibration with negligible overhead (1-2 seconds per layer).

Methodology

Defines a surrogate metric Z that includes cross-terms for expert pairs, capturing cooperation ignored by first-order methods.
Constructs an F matrix where diagonal entries measure individual importance and off-diagonal entries measure pairwise cooperation via gate weights and output norms.
Solves the non-convex QP using continuous relaxation and rounding to binary solutions.

Code is available at github.com/awslabs/HOPE, supporting Qwen3, Qwen3-Next, Qwen3.5, and GLM-4.5 models.
Provides a Python API with calibrate, solve, and prune_model functions, generating HuggingFace checkpoints without modifying vLLM or SGLang definitions.
Benefits are most pronounced at 40-50% pruning ratios; performance is comparable to first-order methods at lower ratios (10-20%).
Currently prunes layers independently and does not account for cross-layer interactions or higher-order interactions involving three or more experts.