Batch API isn't suited for a single agent, but it may be useful for multiple agents

Anthropic's Batch API discounts tokens by 50%, but it's an asynchronous path that can take up to 24 hours to process.

The experimental harness batching-harness wraps every turn as a batch of 1, polls until completion, and then continues the tool loop. It was fitted with a rich-based terminal UI, sandbox-runtime, and a /stats panel comparing the cost of the batch path versus the synchronous path, to see how it actually felt in practice.

In a parallel=1 environment, the advantage of batching nearly disappeared. Having to wait 90-120 seconds per turn stretched a 5-turn agent loop to around 10 minutes, which was too slow for a conversational agent.

The observations were also unexpected.

• Haiku batches tended to take longer than Sonnet or Opus batches.
• The hypothesis is that Haiku's synchronous path is so fast that there's little idle time left for the batch scheduler to step in.
• So the batch path is actually better suited to larger, slower models.

There are specific cases where batching actually pays off.

• Work that doesn't care about latency, like overnight evals or scheduled audits
• Running multiple sub-agents simultaneously
• Pooling requests from multiple harnesses into an actual N-wide batch

In these cases, batch discounts and prompt caching work together, and leveraging the 1-hour cache window can amplify the effect further in asynchronous workloads.

The next step is to add batch-awareness to the local proxy LunaRoute. By simply changing ANTHROPIC_BASE_URL, existing harnesses like Claude Code or Codex don't need to know whether batching is happening at all, and the proxy chooses between the synchronous path and the batch path per request, returning responses through the same interface. The conclusion is simple: the unit of batching isn't one turn from one user, but requests from multiple agents that the proxy can bundle together to send.