Harvard Study: AI Outperforms Doctors

In a Harvard-led study, OpenAI's o1 preview surpassed the level of expert physicians on emergency department care tasks.

The study evaluated 76 emergency room cases from a Boston hospital, assessing judgment at 3 stages:

• Initial triage immediately upon arrival
• The point of first physician contact
• The point of admission to the ward or ICU

Two physicians conducting blind evaluations found that the model performed equal to or better than specialist-level physicians at each stage.

It also showed strength on rare diseases, complex cases, and NEJM cases from Massachusetts General Hospital, and on management reasoning tasks such as antibiotic use, goals of care, and end-of-life conversations, it outperformed both existing AI models and humans using the latest Google search.

However, the input was limited to text-based data, and there is a limitation in that actual clinical care must also handle imaging, test results, and vital signs together. The researchers emphasized that a more realistic direction would be for AI to assist with triage and second opinions rather than replace physicians, and noted that controlled clinical trials are needed to verify real-world application.