Systematic Debugging for AI Agents: Introducing the AgentRx Framework

AgentRx is an open-source diagnostic framework for tracing the origin of failures in long, non-deterministic agent execution paths. It automatically generates guarded executable constraints from tool schemas and domain policies, then verifies them step by step to produce violation logs backed by evidence.

The core idea is to find not the last error, but the Critical Failure Step (the first point beyond which recovery is no longer possible). To do this, it uses a four-stage pipeline—trajectory normalization, constraint synthesis, guarded evaluation, and LLM-based judging—to organize both the cause of failure and its classification.

For evaluation, the AgentRx Benchmark was also released. 115 failure trajectories collected from τ-bench, Flash, and Magentic-One were manually annotated, and a 9-category failure taxonomy was compiled using a grounded-theory approach.

The taxonomy distinguishes the following types of failure:

• Plan Adherence Failure: failing to follow the plan or adding unnecessary actions
• Invention of New Information: fabricating facts not present in the trace or tool output
• Invalid Invocation: incorrect tool calls, missing arguments, schema violations
• Misinterpretation of Tool Output: misreading the tool output
• Intent–Plan Misalignment: a mismatch between user intent and the plan
• Under-specified User Intent: unable to proceed due to insufficient information
• Intent Not Supported: the request cannot be performed with the current tools
• Guardrails Triggered: execution blocked due to safety or access restrictions
• System Failure: connection issues or endpoint outages

In experiments, failure localization improved by 23.6% and root-cause attribution by 22.9% compared to existing prompting-based baselines. The framework, dataset, and paper are all publicly available for use in agent failure analysis and reproducible debugging research.