Gaia2 and ARE Released for Agent Evaluation

HuggingFace has released Gaia2, a successor to the existing GAIA benchmark that can analyze more complex and realistic agent behavior, along with ARE (Meta Agents Research Environments), its execution environment.

Gaia2 moves away from the existing Read-only approach and is a Read-and-write-centered benchmark in which agents directly modify and interact with information. It evaluates the following capabilities through 1,000 new human-crafted scenarios:

• Execution and tool use: Following multi-step instructions and utilizing APIs
• Ambiguity handling: The ability to clarify conflicting requests
• Adaptability and noise tolerance: Responding to environmental changes and API failures
• Temporal reasoning: Performing tasks with time constraints
• Agent-to-agent collaboration: Direct communication between agents

ARE, released alongside it, provides a Smartphone mockup environment that agents can use. This environment includes tools similar to real apps, such as email, calendar, contacts, and file system, and all of the agent's interactions are recorded as Structured traces that can be exported in JSON format. This allows developers to debug the agent's thought process, tool calls, response latency, and more in detail.

The Gaia2 dataset is provided under the CC BY 4.0 license, and the ARE framework is provided under the MIT license.