Ai2 (AllenAI) releases ToyBlocks, a robotic experimentation environment

Ai2 (AllenAI) announced ToyBlocks, an open experimentation environment for analyzing the causes of robotic learning failures and generating simulation data
Achieved a data generation speed of up to 24 times real-time on a single GPU and provided an open dataset for the Franka FR3 robot
The model configuration of Molmo2-4B-tb (high-level planning) and ManiFlow-tb (low-level actions) enables the separate evaluation of planning and action performance
Recorded a full task success rate of 50.0% on the real robot (Franka FR3) compared to 55.5% in simulation, with individual skills achieving an 81.1% success rate

Experimentation Environment and Data

The environment is configured to evaluate visual interpretation, exploration, planning, and precise manipulation using letter/picture blocks and single-color blocks
Utilizes PDDL and TAMP to automatically generate task and action sequences, outputting data in LeRobot v3 format
Includes only successful episodes, resulting in survivorship bias; recovery actions after failure are not included

Molmo2-4B-tb selects the next skill and target coordinates, while ManiFlow-tb outputs arm control and skill completion signals
Recorded a 97.5% success rate with a low-level oracle combination and a 55.5% success rate with a ManiFlow combination under Easy conditions (letters visible)
In agent experiments utilizing Claude Opus 5.5, 63% of 3-letter word exploration tasks resulted in correct tower creation or a determination of impossibility

The Wan 2.2 world model achieved an object position error of 1.0cm and a result prediction accuracy of 99%
When transferring from simulation learning to the real robot, achieved an 81.1% success rate for individual skills and a 50.0% success rate for full tasks without additional learning
Conclusions regarding equivalence between simulation and reality are limited due to the low number of real robot trials (30)