Running 70B Model Inference on a Single 4GB GPU with AirLLM

AirLLM is an open-source project that claims it reduces the memory required during large language model inference, enabling a 70B-class model to run on a single 4GB GPU without quantization, distillation, or pruning.

The core idea is to stream layers sequentially instead of loading the entire model into GPU memory. For sparse MoE models, rather than loading all experts at once, experts are processed one at a time to further reduce memory usage.

The execution examples presented by the repository are as follows.

• Llama 3.1 405B: 8GB GPU
• DeepSeek-V3 671B: about 12GB GPU
• Kimi K3 2.8T: under 4GB GPU

The project provides a Python package, a Quickstart guide, configuration options, macOS support, example notebooks, and an FAQ. However, the figures presented are claims from the repository, and actual processing speed and total model storage/CPU memory requirements should be verified separately.