AI BriefingBigMoeOnEdge enables MoE models larger than device RAM to run on phones and laptops via automatic weight placement
·2026.11.08 18:19
Key point
Built on llama.cpp, the tool automatically allocates model weights to VRAM, RAM, or flash storage to maximize prefill and decode speeds on heterogeneous hardware.
Details
- BigMoeOnEdge is a new tool built on llama.cpp that runs Mixture of Experts (MoE) models larger than the device's total memory by automatically placing weights on VRAM, RAM, or flash storage.
- The system uses an --auto planner to measure hardware capabilities (RAM, VRAM, disk speed) and model routing, assigning frequently used experts to fast memory and rare experts to flash to optimize speed without changing output quality.
- Benchmarks show a 16GB Apple silicon laptop running Qwen3.6-35B-A3B (22GB model) at 18-24 tok/s decode speed, while a desktop with 24GB GPU and 96GB RAM achieves 70 tok/s on Qwen3.8-Flash-Next 125B.
- The tool supports Android, iOS, macOS, Windows, and Linux with backends for CPU, CUDA, ROCm, Vulkan, Metal, OpenCL, and Hexagon NPU, and includes an OpenAI and Anthropic compatible server for agent integration.
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.
Related
gpu-unlock Linux module doubles FP/BF16 instruction rate on Nvidia 3000 series GPUs