Apareto loading plugin for vLLM cuts MoE VRAM usage to 42% with minimal accuracy loss
·2026.11.16 07:12
Key point
The proof-of-concept plugin uses a trained network to dynamically load only top-ranked experts, achieving 42% VRAM usage on SWE-bench Multilingual.
Details
- Introduces Apareto loading, a dynamic VRAM reduction method for Mixture-of-Experts (MoE) models that complements static quantization.
- On Qwen3.6-35B-A3B with NVFP4 experts, the method achieves scores within 2 percentage points of the full model at a 42% VRAM budget on SWE-bench Multilingual and 49% on LiveCodeBench v6.
- A small trained network ranks experts by value after the prefill phase, loading only those that fit the VRAM budget.
- The release is a proof-of-concept code and vLLM plugin, not a production implementation, as the ranking network was trained specifically for the two tested benchmarks.
- Code and reproduction instructions are scheduled for release on GitHub at alphapareto.
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.