The Efficient Frontier of LLM Inference (6-minute read)

In LLM inference, the efficient frontier refers to the maximum level of intelligence achievable for a given cost or model size. In inference engineering, this is primarily expressed as a tradeoff between latency and throughput, and the techniques for managing it fall into two categories: moving to a specific point on the frontier or pushing the entire frontier outward.

Tradeoff Management Techniques

In real-world deployment environments, it is necessary to consider irregular frontiers where parameter changes significantly impact performance. Adjusting batch sizing is the most fundamental tradeoff; increasing the batch size boosts throughput but worsens per-user latency. Regarding parallelization strategies, Tensor Parallelism (TP) is effective for reducing latency, while Attention Data Parallelism (ADP) is effective for improving throughput. Additionally, quantization can enhance serving efficiency without quality loss, showing notable performance improvements particularly with microscaling formats like MXFP4 and NVFP4.

Frontier Expansion Techniques

Techniques that broaden performance limits through optimizations across the hardware and software stack are also important. Kernel optimization and runtime improvements enhance CUDA kernel and forward pass efficiency, reducing token generation resources. Speculative decoding has evolved from small-batch beginnings to latest techniques like EAGLE-3 and DFlash, significantly reducing latency in predictable tasks such as code generation. Finally, P/D disaggregation separates the prefill and decode stages into dedicated workers, adjusting resource ratios according to traffic characteristics to maximize throughput while maintaining or improving latency.