Edge0 Runs 35B MoE Model on 3GiB Memory via SSD Streaming

Developer Edge0 released an SSD expert streaming inference framework. This technology was developed to efficiently run large-scale MoE (Mixture of Experts) models in the limited unified memory environment of Apple Silicon.

Key Technologies and Architecture

• SSD Expert Streaming: Stores model weights on SSD (mmap) and loads them to the GPU only when needed. Uses LRU cache and fixed slots to minimize storage latency.
• Prerouter: Predicts the routing of the next layer one step ahead to hide storage access latency. This prediction module is trained and deployed alongside the model, improving decode throughput by up to 59%.
• Recover-LoRA: Applies LoRA adapters distilled from an fp16 teacher model to restore quality degradation caused by 4-bit quantization.

Performance and Benchmarks

• Memory Efficiency: The edge0-35b model uses 2.9 GiB of peak active memory, and the edge0-8b model uses 1.0 GiB. This represents a very low memory footprint compared to full resident approaches.
• Throughput: On a Mac mini M4 Pro (24GB), edge0-35b records decode speeds of 14.9~17.7 tok/s, and edge0-8b records 23.9~25.3 tok/s.
• Quality: On the OpenCompass benchmark, edge0-35b (int4) scores 79.2, averaging 3.9 points lower than the original fp16 model, maintaining high quality despite 4-bit quantization.

Deployment and Limitations

• Currently supports only the MLX backend and is exclusive to Apple Silicon (M1-M4). CUDA support has not yet been implemented.
• Distributed under the Apache-2.0 license, providing OpenAI-compatible endpoints. Korean language capabilities have not been verified.