Mapping Agent Failures

VAKRA is a tool-grounded executable benchmark that simulates an enterprise environment, evaluating how reliably agents reason and act within real execution flows rather than simple question answering.

The core scale is as follows.

• 8,000+ locally hosted APIs
• Real DBs across 62 domains
• Complex tasks of 3-7 steps under natural language tool-use constraints

The benchmark is divided into 4 capability axes.

• Capability 1: API chaining using BI APIs

  • 2,077 instances, 54 domains
  • Uses SLOT-BIRD / SEL-BIRD tools
  • Requires chaining 1-12 tool calls to reach the correct answer
  • Each instance must first initialize the data source via get_data(tool_universe_id=...)

• Capability 2: Tool selection using dashboard APIs

  • 1,597 instances, 17 domains
  • Centers on domain-specific API selection based on REST-BIRD
  • Number of tools per domain ranges 6-328, averaging 116
  • Shortlisting is required due to OpenAI API's 128 tool limit

• Capability 3: Multi-hop reasoning using dashboard APIs

  • 869 instances, 38 domains
  • Requires 1-5 hops of logical reasoning per question

• Capability 4: Multi-hop, multi-source reasoning and policy adherence

  • 644 instances, 41 domains
  • Includes multi-source problems requiring both API and document search
  • Also demands conversational multi-turn settings and adherence to tool-usage policy
  • Example: hops must match allowed sources as in API - RAG - API
  • Decontamination is applied during data generation so that necessary information remains in only one source

The key point is that, in an environment one step more realistic than existing benchmarks, it reveals where tool selection, multi-step reasoning, and policy adherence collapse simultaneously. The authors report that various models showed generally low performance on VAKRA, and they analyze failure modes for each specific task.