FreeToken Serves 284B MoE Model on RTX 5090

The FreeToken paper, co-authored by 11 researchers from Berkeley, UT Austin, and UC Berkeley, has been released. This research presents an edge serving system designed to address the problem that while open-weight models are publicly available, the hardware barriers to running them remain high.

Existing edge inference engines such as llama.cpp, Ollama, and KTransformers fail to fully leverage the sparsity of MoE (Mixture-of-Experts) models, leaving bottlenecks in expert transfer during the prefill stage and cache misses during the decode stage unresolved. To address this, FreeToken applies three core principles: Bandwidth-Adaptive Execution, Semantic-Aware Caching, and Elastic Edge Resource Management.

Key results include:

• Serving the DeepSeek-V4-Flash (284B) model at 21.5 tok/s on an RTX 5090 (32GB). This outperforms llama.cpp (14.7 tok/s) and KTransformers (10.2 tok/s) on the same hardware, while Ollama was unable to serve the model.
• Serving the Qwen3.6-35B model at 39.3 tok/s on an 8GB RTX 4060 laptop, exceeding the central decode speed of Codex in production environments (33 tok/s).

By treating GPUs, CPUs, host memory, and interconnects as a unified inference platform, this system provides a foundation for running frontier-grade AI models locally on consumer hardware, rather than requiring data-center-grade GPU clusters.