WASTE Inference Engine

WASTE is an open-source inference engine designed to run models locally that are much larger than host memory. Its first supported model is Kimi K3, a 2.78 trillion-parameter Mixture-of-Experts model, converting the original 1.42TB checkpoint into a 982GB container and running it on a MacBook Pro with 64GB of unified memory. It runs the full model, not a distilled, pruned, or otherwise reduced variant.

Kimi K3 selects only 16 of 896 experts per token. WASTE keeps the model trunk resident in memory, places expert weights on NVMe storage and reads them as needed, and uses the remaining RAM as a limited expert cache.

In this architecture, the key is not loading the entire model into memory but reading the active expert data fast enough. K3 accesses roughly 17GB of expert data per token, so storage bandwidth and cache behavior determine performance.

Per-expert matrices are laid out contiguously so they can be loaded with a single positioned read, the OS page cache is bypassed, and the engine manages the expert cache itself. The runtime is written in C with no external inference dependencies, while Python is used only for conversion and validation.

The components currently provided are as follows.

• An embeddable C API
• A CLI client
• An OpenAI-compatible HTTP server
• ARM and x86 CPU implementations
• macOS, Linux, and Windows support
• Text and image input for Kimi K3

When running K3 on 64GB of memory, example performance is about 0.32 tokens/sec, which is still slow for conversational applications. In contrast, Kimi-Linear 48B, at a 19GB container size, recorded about 8.9 tokens/sec on the same system.

During development, small-scale tests that reproduced actual usage patterns were performed before costly operations. Since a 1.42TB source model download and roughly 1TB of conversion work are required, storage benchmarks were also written separately to reflect WASTE's non-sequential expert access pattern.

As a result, external SSDs recorded about 0.94GB/s, while internal SSDs recorded 12.78GB/s. The explanation given is that typical sequential-read performance alone made it hard to identify WASTE's actual bottleneck.