A General Goal-Conditioned Minecraft Model

Pantograph proposes a new training approach that leverages internet-scale video data to address the data scarcity problem in robotics. Previously, goal-directedness was taught during the post-training stage, but this model learns goal-directed behavior starting from the pretraining stage, significantly improving its ability to achieve complex goals.

Key features are as follows:

• Pan model: A 4B-parameter model that performs a variety of tasks including combat, exploration, platforming, and structure building.
• Hindsight Relabeling: By setting later scenes in a video as the goal, the model learns goal-directed behavior without a reward function.
• Training data: The model was pretrained on approximately 500,000 hours of Minecraft gameplay video, then post-trained on 2,000 hours of action sequence data.
• Generality: Because pretraining was conducted using only state information without action information, the model shows strong generalization performance even in new environments.