WARP Runs K3 on 64GB MacBook

WARP (Weight-Aware Runtime and Paging) is a C-based inference engine that avoids loading the full weights of MoE models into RAM, keeping only shared weights and selected experts in memory.

• Target Model: 2.78-trillion-parameter Kimi K3
• Memory: Approximately 29.19GB for a 4K context
• Storage: Approximately 982GB for the converted model
• Performance: 0.45–0.62 tokens per second on an M5 Pro · 64GB MacBook Pro
• Inference Dependencies: No BLAS, Python, or CUDA required beyond libc and pthreads

Expert weights are read directly from NVMe, overlapping computation with disk reads, while remaining RAM is used as an expert cache. It combines 3-bit residual vector quantization with 4-bit and 8-bit quantization for shared weights, and also leverages Kimi K3's compressed KV cache.

Increasing the cache size indiscriminately causes performance to drop sharply. When the expert cache was set to 17.32GB, performance was 0.63 tokens/sec, but at 23.32GB or higher, host memory pressure caused page faults, dropping performance to 0.07–0.09 tokens/sec.

Reducing the number of experts per token from the default 16 to 8 increases speed by 1.49x, but increases KL divergence to 0.037. Disk performance also has a significant impact: unlike the internal SSD's throughput of 12.78GB/s, USB external storage reached only 0.94GB/s.

The project is still in an experimental stage with unstable formats and APIs, making it more suitable for local verification, batch processing, and inference architecture research of large-scale MoE models rather than interactive services.