Running Kimi K3 at 0.50 tok/s with 29GB RAM

WASTE (Weight-Aware Streaming Tensor Engine) is an LLM inference engine written in C that reads only the necessary weights from disk for Mixture-of-Experts models without requiring separate runtime dependencies.

After converting the open-weight Kimi K3 (2.78T parameters) into a 982GiB container, the following performance was confirmed on a 64GB MacBook Pro:

• Minimum RAM: 29.05GiB
• Actual memory usage during execution: 46.24GB
• Speed: 0.49~0.54 tok/s
• Time to generate test sentence: approx. 31 seconds
• Original configuration without model reduction, pruning, or distillation

Kimi K3 activates only about 4% of total parameters per token. WASTE keeps the frequently used model trunk in memory, streams selected experts directly from disk, and utilizes the remaining RAM as a bounded expert cache. The storage format is structured to fetch each expert with exactly one read, and overlapping expert reads with computation yielded approximately 1.6x improvement.

Verification against a PyTorch reference for each layer reported a final logits difference of 3.6e-06 and a vision tower difference of 2.3e-06. However, during the decode phase, disk reads accounted for 55% and computation for 27%, indicating that current performance is very slow. Faster disks or more RAM were suggested as the primary factors for improvement rather than further optimization.

Using the same method, Kimi-Linear 48B ran at 10.7 tok/s with a 19GiB container and 1.87GiB RAM. While WASTE is currently optimized for Kimi K3, it serves as an inference infrastructure example demonstrating that ultra-large MoE models can be run locally on a single consumer-grade system without server-class memory.