NVIDIA Releases AIPerf, an LLM Inference Benchmarking Tool

NVIDIA has released AIPerf, an LLM inference benchmarking tool, as the designated successor to the existing GenAI-Perf. The tool is distributed under the Apache 2.0 license and can be installed via PyPI.

Solving Client-Side Bottlenecks

Existing tools (such as curl and asyncio) suffered from client-side bottlenecks due to the Python GIL and single-process limitations, leading to measurements of client performance rather than server performance. AIPerf resolves these issues by adopting a multi-process architecture using a ZeroMQ message bus, which separates the control, data, and analysis planes. Workers are responsible only for sending requests, while the Record Processor calculates TTFT (Time to First Token) and ITL (Inter-Token Latency) in parallel in a separate process.

Key Features and Metrics

• Inference Model Support: Unlike GenAI-Perf, AIPerf distinguishes inference tokens to separately measure TTFT (all first tokens), TTFO (first output token), and OSL (inference + output).
• Load Simulation: Supports various arrival patterns such as Poisson and Gamma, as well as concurrency bursts, to reproduce real-world traffic variability.
• Goodput Measurement: Aggregates only requests that satisfy SLOs (e.g., TTFT < 100ms) to improve the accuracy of capacity planning.
• Trace Replay: Allows the replay of actual production traces such as Mooncake, Bailian, and BurstGPT to analyze metrics like KV cache reuse rates.

Ecosystem Expansion

This is a collaborative project involving AWS, CoreWeave, Baseten, and Pinterest, rather than a standalone NVIDIA project. It includes features required for enterprise environments, such as a Kubernetes operator and integrations with W&B and MLflow.