GroundedPlanBench: Spatially Grounded Long-Horizon Task Planning for Robotic Manipulation

Existing VLM-based robot planners have handled what to do and where to do it separately, and on long-horizon tasks they easily collapsed due to ambiguity and hallucination in natural language plans. To verify this, GroundedPlanBench consists of 1,009 tasks created across 308 scenes in real robot manipulation environments, evaluating action planning and spatial grounding together.

Each task is decomposed into four basic actions—grasp, place, open, close—and each action is linked to a specific location within the image. Along with explicit instructions, it also includes implicit instructions such as "clean up the table," to examine how consistently a model plans on long-horizon, compound tasks.

On the training data side, the V2GP (Video-to-Spatially Grounded Planning) framework was proposed. It locates interaction moments in robot demonstration videos using gripper signals, has a multimodal model describe the manipulation target, and then tracks objects with Meta's SAM3 to automatically generate grounded plans that include grasp locations and place locations. This process produced 43K grounded plans, composed by length as 34,646 (1–4 actions), 4,368 (5–8 actions), and 4,448 (9–26 actions).

In evaluation, Qwen3-VL was used as the base model, comparing performance before training against performance after fine-tuning with V2GP. In the disaggregated approach, GPT-5.2 or Qwen3-VL-4B first generated a natural language plan, and Embodied-R1 then spatially grounded it, but in scenes with many similar objects, such as napkins, ambiguous expressions caused cases where the wrong instance of the same object was linked.

In contrast, grounded planning handled planning and grounding together within a single model, operating more stably in complex real-world environments. In particular, improvements were observed in both Task Success Rate (TSR) and Action Recall Rate (ARR), and when Qwen3-VL-4B/32B were trained with V2GP, consistent performance gains were confirmed across both explicit and implicit instructions.

The key point is that for robots performing long-horizon tasks, simply producing good natural language plans is not enough—actions and locations must be precisely tied together at the same time. The authors also see a promising future direction in combining this with a world model that includes predicting the next state.