Multiverse Publishes Paper on LLM Block Removal Based on Ising Optimization

Multiverse Computing has released a new LLM compression technique that redefines Transformer block removal (depth pruning) as a physical model known as Ising glass. This study sets the retention or removal of each block as a binary variable and designs an energy function based on the Hessian matrix generated through the second-order Taylor expansion of the loss function. This energy serves as a strong proxy for predicting the benchmark performance of the pruned model, enabling the efficient evaluation of numerous candidate combinations without actual benchmarking.

Core Principles and Efficiency

The researchers transformed this problem into a QUBO (Quadratic Unconstrained Binary Optimization) form and utilized quantum-inspired and classical optimization solvers (such as tabu search) to find low-energy states. This approach can be executed rapidly on a single GPU even for large-scale models and is compatible with Multiverse's existing compression stack (quantization, SVD, etc.).

Performance Results

• Llama-3.3-70B-Instruct: Achieved an MMLU score of 76.9 when removing 40 out of 80 blocks (50% compression), an improvement of approximately 23 percentage points over the existing baseline (54.0).
• Qwen3-14B: Demonstrated competitive performance compared to existing methods even when removing 12 blocks.
• Hybrid Model Application: Proved applicability without retraining on NVIDIA-Nemotron-3-Nano (a hybrid of Mamba2, Attention, and MoE).

Features and Implications

Since energy is not a perfect indicator of performance, a single ground state does not always yield the best model. Therefore, the researchers propose exploring not only the ground state but also low-lying excited states to find the optimal pruning configuration. The code has been open-sourced on GitHub.