OpenAI's o1 Outperforms Triage Doctors in ER Patient Diagnosis Accuracy, 67% vs. 50-55%

Harvard researchers compared OpenAI o1 with human doctors based on electronic medical records of 76 emergency room patients at a Boston hospital.

• o1 recorded 67% on correct or very close diagnoses, outperforming triage doctors' 50-55%.
• Under conditions with more information, accuracy rose to as high as 82%, while human specialists recorded 70-79%, though the difference was not statistically significant.
• In treatment plan evaluation using 5 clinical case studies, o1 scored 89%, while 46 doctors scored 34%.

The researchers emphasized that this experiment was limited to information that could be conveyed in text. Since patients' appearance, distress, and non-verbal cues were excluded, the results suggest AI could be useful as a second-opinion tool rather than a replacement for doctors. However, it was also pointed out that safety, liability, and bias across patient groups still require further verification.