Limitations of Pixel Metrics in World Model Evaluation and the Diagnostic Tool worldproof

worldproof is an open-source tool that diagnoses where and why World Models, which predict future frames, fail in their predictions. Unlike existing benchmarks that simply measure task success rates, it analyzes the causes of errors by comparing prediction results with Ground Truth data and Physical Invariants.

Key Insight: The Discriminative Problem of Pixel-Based Metrics Analysis of real robot video data confirmed that pixel-based metrics such as SSIM and PSNR fail to correctly rank performance between models in certain situations. When the prediction Horizon lengthens but errors do not increase and remain flat, performance differences between models cannot be distinguished.

Experimental results observed the following three intervals depending on the model's prediction step:

• Perfect Match Interval (Steps 1–3): All models record nearly perfect scores, with no discriminative power between models.
• Discriminative Interval (Steps 4–24): The metric drops sharply, making this the only interval where performance differences between models are clearly revealed.
• Loss of Correlation Interval (After Step 28): The metric remains at its lowest point and fluctuates randomly. This indicates that predictions have become completely unrelated to actual data.

These results suggest that in world model evaluation, it is crucial to verify not only pixel errors but also whether the evaluation setup itself possesses the ability to distinguish between models.