OpenAI o1 Outdiagnoses ER Doctors in New Study

AI Outperforms Emergency Room Doctors in Diagnostic Accuracy

A new study has found that OpenAI's o1 reasoning model correctly diagnosed 67% of emergency room patients — significantly outperforming human triage doctors, who achieved accuracy rates of just 50-55%. The findings, reported by Robert Booth in The Guardian, are being hailed by researchers as evidence of a 'profound change in technology that will reshape medicine.'

The AI system worked with the same limited information available to triage physicians: electronic health records and a few brief sentences from nurses describing patient symptoms. Despite this constrained input, o1 demonstrated a marked advantage in diagnostic accuracy.

How the Study Worked

The research tested OpenAI's o1 model — the company's advanced reasoning-focused large language model — against real-world emergency department scenarios. Rather than giving the AI access to comprehensive patient histories, lab results, or imaging, researchers deliberately limited its inputs to mirror the conditions faced by triage doctors at the front line of emergency care.

Triage physicians typically make rapid assessments under intense time pressure, often with incomplete information. They rely on brief nursing notes, electronic health records, and their own clinical experience to prioritize patients and form initial diagnostic impressions. The study essentially asked: can AI do better with the same sparse data?

The answer, at 67% versus 50-55%, was a clear yes.

Why This Matters for Emergency Medicine

Emergency departments worldwide face chronic staffing shortages, overcrowding, and burnout among physicians. Misdiagnosis at triage can lead to delayed treatment, unnecessary tests, or — in worst-case scenarios — patient harm. Even a modest improvement in early diagnostic accuracy could have significant downstream effects on patient outcomes and resource allocation.

The 12-17 percentage point gap between AI and human performance is particularly notable because it emerged under realistic constraints. This was not a case of AI being fed pristine, comprehensive data while doctors worked with scraps. Both operated under the same informational limitations that characterize the chaotic reality of emergency medicine.

Researchers involved in the study described the results as marking a 'profound change in technology that will reshape medicine,' suggesting this is not merely an incremental improvement but a potential paradigm shift in how emergency departments could operate.

The Role of Reasoning Models

OpenAI's o1 model differs from its predecessors like GPT-4o in its emphasis on deliberate, step-by-step reasoning. Rather than generating immediate responses, o1 'thinks' through problems methodically — a characteristic that appears particularly well-suited to medical diagnosis, where connecting disparate symptoms to underlying conditions requires logical inference.

This study adds to a growing body of evidence suggesting that reasoning-optimized LLMs may be especially powerful in clinical settings. Previous research has shown strong AI performance on medical licensing exams and clinical vignettes, but real-world diagnostic studies using actual patient data carry significantly more weight in the medical community.

Limitations and Caveats

Despite the promising results, significant questions remain. A 67% accuracy rate still means one in three patients would receive an incorrect initial diagnosis from the AI. In emergency medicine, where life-threatening conditions demand rapid and correct identification, a 33% error rate is far from acceptable as a standalone diagnostic tool.

Additionally, triage doctors do more than diagnose — they assess patient acuity, make disposition decisions, and exercise clinical judgment that encompasses body language, patient demeanor, and subtle physical signs that no electronic record can capture. AI cannot yet replicate the full scope of a physician's bedside assessment.

There are also concerns about liability, patient trust, and the 'black box' nature of AI decision-making in high-stakes medical environments. Regulatory frameworks for AI-assisted diagnosis remain in early stages across most Western healthcare systems.

The Road Ahead

The study's findings are likely to accelerate conversations about integrating AI into emergency department workflows — not as a replacement for physicians, but as a decision-support tool that could flag potential diagnoses and reduce cognitive load during high-pressure triage situations.

Several health systems in the US and UK are already piloting AI-assisted clinical tools, and results like these will likely drive further investment. OpenAI, Google DeepMind, and other major AI labs have all signaled that healthcare is a priority vertical for their models.

As AI reasoning capabilities continue to improve — with models like OpenAI's o3 and competing systems from Anthropic and Google on the horizon — the gap between AI and human diagnostic performance could widen further. The question is no longer whether AI can assist in medical diagnosis, but how quickly healthcare systems will adapt to integrate it safely and effectively.

For now, the study serves as a powerful data point: in at least one critical medical scenario, AI is already outperforming the doctors it may one day assist.