Benchmarking Inference Engines for Agentic Workloads

Existing LLM inference benchmarks have focused on one-shot prompt/decode, but agentic apps operate by calling tools across multiple turns. Because of this, KV cache management, scheduler pressure, and long-tail distributions become key variables that determine performance.

Based on traces obtained from production post-training runs and deployments, the maker released 3 workload profiles and an open-source benchmarking harness that replays them. The targets are agentic coding (based on SWE-Bench Verified), repository-exploration-style code QA, and office work that handles documents, spreadsheets, and slides.

The observed traces showed an average of about 20 tool turns, with assistant responses mostly clustering around 200~300 tokens, tool output around about 500 tokens, and input prompts around about 10k tokens. However, the number of turns, output length, and tool call latency all had long tails, which could stretch to tens or even hundreds of seconds.

Terminology was also clarified. A request is a single /completion or /chat/completion call, a trace is all requests and tool calls in one agent session, and a workload is a collection of multiple traces. Parallel tool calls were recorded merged into one larger prefill request, so as to reflect the engine-level cost and semantics as-is.

Evaluation metrics differ depending on the deployment type.

• Batch: Since minimizing $/token matters, completion throughput per GPU is viewed as the main metric.
• Background: Since SLA compliance matters, end-to-end trace latency and p90/p99 tail latency are examined.
• Interactive agents: TTFAT, the time until the first answer visible to the user, is key, and interactivity, which represents streaming speed, also matters.

They also pointed out that a simple mean trace replay fails to capture high variance and convex cost, and can overestimate actual performance. The core point of this piece is that replaying the full trace distribution is necessary to properly measure scheduling and KV allocation pressure.