vLLM Releases GuideLLM v0.8.0

The vLLM project has released GuideLLM v0.8.0, an LLM serving performance benchmarking tool. The core of this update is measuring goodput—the throughput of requests that meet latency SLOs—instead of simple throughput (RPS), and automatically searching for the maximum concurrency that achieves it.

SLO-based goodput and concurrency search

When a server reaches saturation, there is a region where throughput is maintained but latency spikes. GuideLLM v0.8.0 calculates SLO attainment and request goodput by setting TTFT, TPOT, and E2E latency thresholds along with compliance request ratios. Using the goodput profile, it doubles concurrency until failure, then performs a binary search at the failure boundary to find the optimal concurrency that maintains the target attainment rate (default 0.95). In the documentation example, at concurrency 16, goodput was 24.6/s (100% attainment), but at concurrency 32, although the number of requests was similar, goodput dropped by 22% to 19.3/s (79.6% attainment).

Agent trace replay support

To reproduce complex agent loads, it supports trace replay in WEKA and OpenTelemetry (OTEL) formats. The WEKA format handles session-based conversations including sub-agents and tool calls, providing SemiAnalysis's Claude Code trace dataset as an example. The OTEL format replays only successful LLM calls based on GenAI spans, and via the history option, users can choose between sending all recorded messages (trace) or sending based on real-time responses (runtime). This allows analysis of performance characteristics in real agent environments, such as prefix cache hit rates.

Upgrade notes

There are Breaking Changes when upgrading from v0.7.x to v0.8.0. The default value (sweep) for the --profile argument has been removed and must now be specified. Existing built-in scenarios (chat, rag, etc.) have been deprecated and replaced with seven scenarios in the rhaiis/... format. Additionally, the default worker process has been changed to spawn. In v0.8.1, the use of list-form profile arguments will be deprecated, so using --override is recommended.