Epoch AI: Frontier Models Excel at Defined Tasks but Fail at Open-Ended Work Automation

Epoch released Epoch Automation Reports, evaluating six models—Claude Fable 5.1, GPT-6 Astra, Grok 4.6, Gemini 3.8 Flash, Kimi K3, and Qwen 3.8 Max—on real-world business automation tasks drawn from Epoch's own work. The central finding is that while closed-weight models like Fable 5.1 and GPT-6 Astra are reliable for well-defined tasks such as coding, computational analysis, and computer use (e.g., successfully porting content to Substack), they fail at open-ended tasks requiring research judgment, preventing full automation.

Performance and Limitations

Fable 5.1 and GPT-6 Astra led in aggregate scores and demonstrated strong computer use capabilities. However, they struggled with Graphic Design and Data Insight Generation tasks that required subjective judgment and adherence to implicit conventions, such as Epoch's visual style and audience preferences. Open-weight models like Kimi K3 and Qwen 3.8 Max showed higher error rates even in defined tasks, indicating a larger gap between benchmark scores and actual capability than previously assumed. For instance, Kimi K3 produced a factually inaccurate data insight due to a filtering error, while closed-weight models did not make such factual errors in data insights.

Research Judgment Failures

A critical weakness identified was the models' inability to design valid experiments. In a test where GPT-6 Astra was asked to investigate AI agent learning failures, it proposed a pilot experiment with a 4096-token input budget. This constraint caused 61 of 280 responses to be truncated, leading the model to infer rules from incomplete data. Although Astra recognized the error and increased the budget to improve performance, it misinterpreted the initial failure as "sensitivity to the acquisition budget" rather than an experimental setup flaw, treating the flawed results as key findings. This pattern of treating flawed results as significant findings was observed across models, highlighting a lack of research judgment.

Convergence and Standards

Models often converged on similar ideas for open-ended tasks, lacking diversity. For example, Fable 5.1, Gemini 3.8 Flash, and Grok 4.6 all chose the same topic for a polling data insight. Additionally, models failed to pick up Epoch's standards despite ample reference material, missing implicit conventions like visual style and topic relevance. The study concludes that while AI can handle well-defined tasks, it cannot yet replace workers at Epoch due to gaps in judgment, pattern learning, and experimental design.