AirLLM Supports 2.8T Model with 4GB VRAM

AirLLM dramatically reduces inference memory usage without quantization or distillation, allowing 70B LLMs to run on 4GB GPUs, Llama 3.1 405B on 8GB, and DeepSeek-V3 (671B) on approximately 12GB. Specifically, for sparse MoE models, it streams only the experts routed by tokens rather than entire layers, enabling the largest open-source model released to date, Kimi K3 (2.8T), to run on less than 4GB VRAM.

Key Updates

• 2026/08: Support for Qwen3.8-27B. A new Dense VL model featuring Gated DeltaNet and Gated Attention, measured at 3.33GB VRAM on an RTX 3090. Requires transformers 5.8+.
• 2026/07: Support for Kimi K3 (2.8T). Measured at 3.72GB VRAM on an RTX 6000 Ada. compressed-tensors, flash-attn, CUDA 12-based torch, and transformers 4.56.x are mandatory requirements.
• 2026/06: v3.0 release. Added support for FP8 models and latest models. DeepSeek-V3 (671B) can run on approximately 12GB, and Qwen3-235B on approximately 3GB.