IBM Releases 'Consistency Gap' Metric for AI Agent Consistency

IBM Research defined a new evaluation metric called Consistency Gap to address reliability issues in AI agents and released a tool called Consistency Analyzer to improve it.

Definition and Problem of Consistency Gap

Existing Mean@k (average pass rate) effectively represents the average performance of an agent but overlooks the possibility of failure when repeating the same task. To complement this, IBM introduced Pass^k (the ratio of succeeding all k times) and named the difference between Mean@k and Pass^k as Consistency Gap.

Results from running a GPT-4.1-based ReAct agent 5 times on the AppWorld test:

• Mean@5: 77.4%
• Pass^5: 53.0%
• Consistency Gap: 24.4 points

This suggests that the issue is not a lack of model capability, but rather inconsistency caused by the flatness of the decision distribution.

Consistency Analyzer and Improvement Effects

The proposed tool is based on the existing ALTK-Evolve pipeline and identifies unstable decision points by performing k resamplings (k=5) at each decision step of a single trajectory without Ground Truth.

• How it works: Requests 5 completions at each decision step to measure output variability and generates guidelines targeting unstable steps.
• Improvement results: Applying the guidelines reduced the Consistency Gap by half, from 24.4pp to 12.0pp.
• Performance enhancement: Aggregate Pass^5 rose from 53.0% to 69.0%, while Mean@5 was maintained or slightly improved (77.4% -> 81.0%) without dropping.
• Generalization: Consistency improvement effects were confirmed on similar tasks and weaker models (gpt-oss-120b), proving it is a reusable pattern correction rather than memorization of specific trajectories.

Practical Implications

When deploying agents, Pass^k should be reported alongside Mean@k, and it should be recognized that the consistency gap widens for more difficult tasks. IBM emphasized that improving consistency is prioritized over using larger models, and the diagnostic process can be performed in production traffic with just one additional LLM call. Related code can be found on GitHub (ALTK-Evolve) and arXiv (2609.08832).