sglang: LLM Inference Engine Powering 400,000 GPUs Worldwide

A serving framework that handles low-latency, high-throughput inference for LLMs and multimodal models, from a single GPU to large-scale distributed clusters. It maximizes the speed of processing repetitive requests with RadixAttention-based prefix caching and a zero-overhead scheduler.

It broadly supports major open-source models such as Llama, DeepSeek, and Qwen, as well as Diffusion and embedding models. It runs on diverse hardware environments including NVIDIA, AMD, Google TPU, and Intel CPU, optimizing performance with FP4/FP8 quantization and Speculative Decoding.

Global companies such as xAI, NVIDIA, and Alibaba have adopted it in production environments, using it to generate trillions of tokens per day. It is also actively used as a rollout backend for model training, integrating with major reinforcement learning frameworks like AReaL and verl.