Real-Time LLM Inference on Standard GPUs: 3,000 tokens/s per Request

The productivity of AI agents is determined not just by model intelligence but by iteration speed. Since an agent's workflow (planning, writing code, testing, fixing) consists of a sequential loop, decode speed per request rather than overall throughput determines user experience and the practical performance of a product.

The main bottleneck in current GPU-based inference is not compute power (FLOPS) but Memory Bandwidth. When batch size is 1, autoregressive decoding is dominated by matrix-vector operations, and the speed at which the model's weights are moved from memory to the processor limits overall performance. Therefore, Memory Bandwidth Utilization (MBU) rather than Model FLOPS Utilization (MFU) becomes the key metric for single-request speed.

To overcome the software-level limitations of existing inference stacks, Kog proposes the following co-design approach:

• Integrating model architecture, runtime, and low-level GPU kernels into a single latency-optimized pipeline
• Maximizing the potential of standard datacenter GPUs through a design optimized for batch size 1

Through this optimization, they demonstrated that even large MoE (Mixture-of-Experts) models can achieve extremely fast decoding speeds of up to 3,000 tokens per second using standard GPUs.