How to Train Robot World Models Without Synthetic Data

F.R.A.N.T.Z. is a method that extracts the scene semantics needed for robot world model training from general monocular video, without synthetic data or domain-specific demonstrations. It combines multiple pretrained models with classical computer vision techniques to generate objects, planes, affordances, hand-arm kinematics, and metric 3D point clouds.

Existing Vision-Language-Action (VLA) models train raw pixels, language instructions, and motor actions end-to-end as one massive model. Approaches like RT-2, Octo, and π0 require millions of robot demonstrations or billions of image-text-action combinations, and rely on simulators, digital twins, and generative video to compensate for the lack of real-world data.

This approach has three structural limitations.

• As synthetic data increases, the sim-to-real gap grows larger.
• The black-box structure with billions of parameters makes it difficult to diagnose the cause of model failures module by module.
• What the industry calls 'zero-shot' is often prompt-level zero-shot that presupposes large-scale pretraining.

What F.R.A.N.T.Z. aims for is data-level zero-shot, never having seen domain data at all. By combining different general-purpose models such as depth estimation, object detection, and hand landmark detection, it interprets a given scene without having learned a specific kitchen or manipulation task.

The core components include monocular depth estimation from MiDaS, trained on internet photos, object detection from YOLO, trained on general-purpose object datasets, and hand landmark detection from MediaPipe, trained on general hand images. Classical computer vision based on OpenCV is combined with these to integrate each module's output into scene-level structure and semantics.

This approach suggests a direction for transferring affordances and prior knowledge of new scenes with just a single human demonstration, rather than millions of additional data points. It could also be combined with the COOL framework, which accelerates OpenCV on AWS Graviton, to pursue cost-efficient scaling.

Ultimately, this leads to a JEPA (Joint Embedding Predictive Architecture)-based world model that predicts semantic representations of a scene instead of predicting the pixels themselves. This is a method that allows the robot to predict future states centered on important semantics such as objects, spatial structure, and action possibilities, rather than reproducing every visual detail.