IBM, UC Berkeley Identify Root Causes of AI Agent Failures

IBM Research and UC Berkeley applied the MAST (Multi-Agent System Failure Taxonomy) methodology to analyze why Agentic LLM systems fail in real-world IT automation environments. The core aim is to solve the 'black box' problem in which existing benchmarks rely solely on a single metric—success rate—leaving 'why' a failure occurred unknown.

The research team analyzed execution logs from ITBench, an IT automation benchmark, and discovered the following key failure patterns.

• Differences in failure patterns by model: Frontier models like Gemini-3-Flash tend to fail locally at specific bottleneck points such as the verification stage, whereas large open models like GPT-OSS-120B experience cascading failure, where a single error leads to a chain of hallucinations.
• Incorrect Verification: This emerged as the strongest predictor of failure across all models. The FM-3.3 pattern, in which an agent declares success without actually checking the result, was frequently observed.
• Termination control issues: The Kimi-K2 model showed a notable tendency to fail to recognize task completion, either terminating prematurely or falling into infinite loops.

Recommendations for building reliable agents include the following.

• Externalize verification: Rather than letting the LLM grade its own output, objective evidence obtained through a Tool should always be required.
• Loop and termination control: Explicit stopping conditions or a finite state machine (FSM) should be implemented outside the model to prevent infinite loops.
• Handling ambiguity: When input values are unclear, agents should be designed to ask the user questions, reducing judgment errors.