Stanford Researchers Propose EXPO-FT for Universal Post-Training in Robotics

Researchers from Stanford, including Perry Dong and Chelsea Finn, argue that robotics is at a stage similar to early language models, where pretraining succeeds but reliable deployment fails due to a lack of standardized post-training recipes. They propose EXPO-FT, an algorithm designed to stabilize reinforcement learning for large pretrained policies by isolating instability in a lightweight editing policy. Unlike traditional RL that modifies large model weights directly, EXPO-FT uses this editing policy to refine outputs from a frontier model, guiding them toward high-value states evaluated by a value function. Successful actions are then distilled back into the base model, allowing it to learn new behaviors while maintaining stability. In experiments with physical robots performing tasks such as routing holiday lights, sinking pool balls, and inserting flowers into bottles, EXPO-FT achieved a 30/30 success rate (100%), significantly outperforming baselines like SFT and HIL-SERL. However, the method currently requires human intervention for defining success, resetting environments, and providing feedback. The authors highlight that scaling this approach requires standardized practices for reward specification, reset mechanisms, and hyperparameter tuning, which are currently missing in the field.