vllm: LLM Inference Engine Based on PagedAttention

A high-performance engine for LLM inference and serving that centers on PagedAttention technology to maximize memory efficiency. Developed at the UC Berkeley Sky Computing Lab, it has grown into a large-scale open-source project with over 2,000 contributors.

It accelerates request processing through continuous batching, chunked prefill, and prefix caching. It supports various quantization methods such as FP8, INT4, and GPTQ, as well as optimized attention kernels like FlashAttention and FlashInfer, to utilize hardware resources efficiently.

It supports over 200 model architectures from Hugging Face, covering a wide range from decoder-only LLMs like Llama, Qwen, DeepSeek-V3, and Mixtral to MoE, multimodal, and embedding models. It allows for flexible deployment environments through various hardware plugins, including NVIDIA, AMD, and Intel GPUs, as well as TPU, Gaudi, and Ascend.

It provides an OpenAI-compatible API server, Anthropic Messages API, and gRPC to facilitate integration with existing systems. It supports distributed inference leveraging tensor, pipeline, and data parallelism, along with streaming output and structured output generation, meeting the LLM serving requirements of production environments.