airllm: Running 2.8T Kimi K3 on 4GB VRAM

Unlike existing methods, this approach significantly reduces memory usage during inference without quantization, distillation, or pruning. It enables running 70B-scale LLMs on a single 4GB GPU card, 405B Llama 3.1 on 8GB, and 671B DeepSeek-V3 on approximately 12GB. Notably, large-scale MoE models such as the 2.8T Kimi K3 can run on less than 4GB of VRAM.

The core of its memory efficiency lies in layer-wise streaming and expert-level loading. Instead of loading all layers at once, it sequentially loads only the necessary parts. For MoE models, it loads only the experts to which tokens are actually routed. Additionally, applying block-wise quantization speeds up inference by up to 3x with negligible accuracy loss.

It supports the latest models, including Qwen3, Llama 3.x/4, DeepSeek V2/V3, Phi-4, and Gemma, through a single AutoModel interface. The 27B-scale Qwen3.8-27B runs on 3.33GB VRAM, and the 235B Qwen3 runs on approximately 3GB. It also supports macOS (Apple Silicon) and CPU inference; simply inputting a Hugging Face model ID automatically detects and initializes the model type.

It is useful for experimenting with or prototyping large models in environments lacking high-spec GPUs. If disk space is limited, you can configure it to delete the original model and retain only the converted model, reducing storage usage by half. Released under the Apache-2.0 license, it can be freely used in commercial projects.