Edge0: Run a 35B MoE Model on 2.5GB RAM

An open-source inference framework that runs a 35B-scale MoE model on approximately 2.5GB of RAM. It streams expert weights from the SSD, and a learned router performs load prediction for the next step to minimize latency. Peak memory is determined by the active expert set rather than the total parameter count.

It operates on the same recipe across four platforms: Mac, iPhone, Android, and Windows. It provides a Python-based MLX backend and native engines for each platform, with 4-bit quantization and Recover-LoRA resulting in an average performance loss of only 3.9 points compared to FP16. On the MMLU-Pro benchmark, it actually recorded scores higher than the base model.

Two model tiers, Edge0-35B-A3B and Edge0-8B-A1B, are distributed together. Each checkpoint includes a trained LoRA adapter and a pre-router head, allowing the inference pipeline to run immediately without separate configuration. On a Mac mini M4 Pro, the 35B model's decode speed was measured at 14.9~17.7 tok/s, with peak memory at 2.9 GiB.

Suitable for developers looking to run large-scale sparse MoE models on consumer hardware. It is released under the Apache-2.0 license, and model files can be downloaded from Hugging Face and ModelScope. The base checkpoint and adapters are placed together in a single directory, making deployment and updates easy.