AsgardBench, a Benchmark for Visually Grounded Interactive Planning

AsgardBench is a benchmark that evaluates how well embodied AI agents revise their plans based on visual information while performing a task. Rather than simply measuring whether a task is completed, it focuses on the ability to re-plan the next action according to state changes observed visually.

Built on top of AI2-THOR, this benchmark places the agent at interactable positions near objects and allows only a restricted set of actions such as find, pickup, put, clean, toggle_on/off. At every turn, the agent proposes a full plan to complete the task, but in practice only the first step is executed, after which the agent receives a new image and a success/failure signal and must re-plan.

The key point is not the difficulty of navigation or manipulation, but whether the agent immediately updates its plan using the observed state. For example, even for the same instruction to 'wash the cup,' the required sequence of actions differs depending on whether the cup is already clean, dirty, or contains coffee. The same applies when the sink is full of other objects—the same instruction requires different execution paths.

The evaluation results clearly show the importance of visual grounding.

• Across multiple models, image input substantially boosted success rates compared to text-only conditions.
• Unlike some prior benchmarks, a simple textual description of failure was not enough; visual information is what makes the difference in performance.
• Even the strongest vision-capable models outperformed text-only agents, revealing that this task demands perception-based reasoning that cannot be substituted by text alone.

Recurring failure patterns were also observed. Agents attempted infeasible actions, repeated the same action in a loop, or misinterpreted subtle visual cues such as clean/dirty or on/off. They also frequently lost track of current task progress across multiple steps, indicating the need for more sophisticated state tracking and plan revision capabilities.

AsgardBench also serves as a diagnostic tool that lets researchers isolate performance differences by varying feedback levels between none / minimal / detailed. The authors believe that stronger visual understanding, more reliable state memory, and learning methods that revise plans mid-task will be the next challenges for embodied agents. The benchmark is open source and available on GitHub.