Anthropic Releases Benchmark for Measuring Drone-Piloting Ability

Anthropic, in collaboration with Andon Labs, announced Project Pilot, which measures whether an AI model can directly pilot a quadrotor drone indoors to detect and track a specific person. The core output of this experiment, Drone-Bench, is a new benchmark for evaluating AI's physical autonomy.

Drone-Bench breaks down an aerial surveillance mission into the following five subtasks and produces a score for each model:

• Localization
• Path Following
• Target Detection
• Tracking
• Reconstruction and Control

This project focuses on measuring AI's ability to handle physical-world tools (robots, drones, etc.) beyond the realm of chatbots. In particular, its purpose is to proactively identify the risks of misuse that can arise when dual-use technologies like drones are combined with AI, and to secure reproducible numerical evidence for this.

The evaluation consists of two layers to ensure statistical significance: a Demonstration using actual drones, and a Evaluation that implements this in software to allow for repeated execution.