OpenEBFM: 1B-Parameter Energy-Based Model Outperforms GPT Baseline on Zero-Shot QA

·2026.10.30 14:22

Key point

The largest 1B-parameter model achieves 44.71% average zero-shot QA accuracy, surpassing the 39.79% of the corresponding GPT baseline.

OpenEBFM separates generation into energy steering and token prediction, sharing a Transformer backbone to optimize latent states toward lower-energy regions.
The largest 1B-parameter model achieves 44.71% average zero-shot QA accuracy on 19 tasks, compared to 39.79% for the matched GPT baseline.
EBFMs show faster improvement than GPT baselines when scaling parameters from 268M to 1B and achieve lower validation BPB with increased training tokens.
Full training stack, pretrained checkpoints, and an EBFM-based chatbot are released as open source with the paper and code available.

This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.

Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.