Janas-LLM Streams MoE Experts from NVMe SSD to Run Models Larger Than RAM

Janas-LLM is an experimental, dependency-free inference engine written in pure C that enables running Mixture-of-Experts (MoE) LLMs larger than available system RAM by streaming expert weights directly from NVMe SSDs. Developed by Maurizio "camauri" Cammalleri, the tool caches frequently used experts in memory while fetching others from disk on a per-token basis, targeting Linux and Windows platforms.

Performance Benchmarks

In tests conducted on a laptop with an Intel Core Ultra 9 185H, 32GB DDR5 RAM, and a 1TB NVMe SSD, Janas-LLM demonstrated superior performance compared to llama.cpp (build 2b18470, Vulkan backend) across multiple models. Notably, for the Qwen3-Next-80B-A3B (48.4GB), which exceeds the system's RAM:

• Janas-LLM: 62 tok/s (prompt) / 24 tok/s (generation)
• llama.cpp: 16 tok/s (prompt) / 2.8 tok/s (generation)

The engine also showed significant gains for other MoE and dense models, such as Qwen3-Coder-Next (22 tok/s vs. 2.2 tok/s for llama.cpp) and Qwen3-30B-A3B (32 tok/s vs. 15 tok/s). An exception was noted for 3-bit quantized Qwen3-30B-A3B, where Janas was slower in prompt reading but three times faster in generation.

Architecture and Features

Janas-LLM operates without external dependencies for its core library, using O_DIRECT for disk access and managing memory aggressively. Key features include:

• Drafting Support: Implements Multi-Token Prediction (MTP) for Qwen3.5/3.6/Next and Gemma 4 models, with draft survival rates between 63% and 89% depending on the model.
• Services: Includes janas-chat for dialogue, janas-server for an OpenAI-compatible HTTP API, and janas-mcp for MCP server integration.
• Resource Management: Users can limit memory usage via --cache <GiB> and disable GPU acceleration with JANAS_GPU=0.

Limitations and Stability

The project is in early development with an unstable API. It pushes system resources to their limits, potentially causing desktop lag or GPU driver resets if swapping occurs. Performance degrades to approximately 75% when other applications like browsers are active, with disk wait times increasing fivefold. The tool is currently validated only on specific Intel hardware; AMD and discrete GPU configurations remain untested.