Pantheon Team Builds 1M-Task UMI Dataset for $10/Hour Using Custom Hardware and Argus Pipeline

Pantheon Team announced the creation of the world's most diverse Universal Manipulation Interface (UMI) dataset, collecting over 1 million unique tasks in just 8 weeks. The project achieved a collection cost of $10 per hour, a substantial reduction from the $60–$150/hour typical for external data purchases. This effort supports the development of general-purpose robotics foundation models by addressing the limitations of existing task-sparse imitation learning datasets.

Collection Strategy and Data Quality

The team employed a multi-pronged collection approach to ensure diversity and robustness. Freeform Collection involved operators performing unscripted tasks with 15 items on a table, leveraging the Argus annotation pipeline to generate dense task labels. This method prioritized high throughput and environmental variety but initially lacked fine-grained long-tail actions. To address this, Scripted Collection utilized the Nomos platform, which combines object catalogs with manipulation primitives to generate millions of distinct, physically valid tasks.

Additionally, the dataset includes Failure Data to train recovery behaviors, capturing episodes of missed grasps and premature drops. In-the-wild Data was collected using tether-free hardware to overcome the environmental constraints of central office spaces. The resulting dataset provides broad coverage of world states and failure modes, unlike previous datasets limited to successful, scripted demonstrations.

Custom Hardware and Infrastructure

Pantheon developed three generations of the UMI Gripper to optimize for mass production and teleoperation. The 3rd generation gripper features minimal assembly, a production cost of $11 per pair (compared to $80–$110 external quotes), and compatibility with off-the-shelf camera modules. Custom firmware enables multi-camera synchronization and metadata attachment without additional sensors.

To handle the massive data volume of 2 PB per month, the team built a custom infrastructure stack:

• Hades: A 20 PB+ bare-metal storage cluster in a data center near the SF office, reducing annual storage costs to $100,000 versus $5.2 million for hyperscale object storage.
• Styx: A 1 PB local storage cluster colocated with operations to mitigate internet bandwidth bottlenecks.
• Pallas: Gaming PCs equipped with RTX 5080 GPUs for video decoding and re-encoding at 1.9x real-time throughput, facilitating daily data transfers of up to 90 TB via private DIA.

Benchmark Performance and Scaling

The dataset significantly outperforms existing public UMI benchmarks in unique task labels. Pantheon recorded 1,089,403 unique tasks between August 3 and September 28, 2026. In comparison, Hy-UMI reported 992 unique labels, HiFi-UMI-2K had 588, and FastUMI-100K had only 30. The team projects that maintaining current pace could yield 12 million unique tasks by 2027, with a linear extrapolation suggesting up to 25.1 million tasks. Future plans include an app-based system to engage thousands of global operators for further data collection.