Microsoft's MAI-DxO scores 83% on hard NEJM cases: what the Science paper shows and what it does not

Microsoft AI says its diagnostic system MAI-DxO answered 83 percent of hard New England Journal of Medicine (NEJM) cases correctly in a simulation, against 36 percent for physicians working alone with reference materials. The study was published in Science on October 22, 2026, and Microsoft AI chief Mustafa Suleyman frames it as a step toward "medical superintelligence." The numbers are real, but the headline compares a software panel with single doctors, in a simulation, using models Microsoft did not build.

Here is what was tested, what the independent reading adds and why the superintelligence label should wait.

What MAI-DxO is and how it was tested

MAI-DxO is an orchestrator: several AI agents that act like a panel of doctors. They propose diagnoses, challenge each other over multiple rounds, ask a simulated patient questions, order tests and weigh each test's value against its cost. It runs on top of existing models; Microsoft tested models from OpenAI, Google, Anthropic, Meta and DeepSeek.

The test is the Sequential Diagnosis Benchmark (SDBench). It presents a case step by step: the clinician, human or AI, starts with limited information and must request tests and ask questions, and is scored on accuracy, the evidence requested and estimated test cost. It uses 304 complex NEJM case proceedings, plus an emergency department set from the MIMIC-IV-ED database. Microsoft has open-sourced the benchmark code, though only select NEJM cases are open, which limits independent reproduction.

The results

Microsoft's page reports:

• NEJM cases, physicians without reference resources: 22 percent against 83 percent for MAI-DxO.
• NEJM cases, physicians with textbooks and the internet: 36 percent against 83 percent.
• MIMIC-IV-ED emergency cases: 47 percent for physicians against 80 percent for MAI-DxO.
• A cost-optimized setup reached 81 percent on NEJM cases at less than half the cost of comparable systems.

Costs are estimates, not realized spending. For context, the first public version of this work, a June 2025 preprint by Microsoft researchers including Suleyman, Dominic King and Eric Horvitz, reported 80 percent accuracy with OpenAI's o3 and 85.5 percent in a maximum-accuracy configuration, against a 20 percent average for generalist physicians. The new Science paper extends that to newer models and an emergency dataset.

What the independent reporting adds

TechCrunch's account makes the gap between headline and mechanism clearer. GPT-5 alone with a short basic prompt scored 78.9 percent on the complex cases at about $8,400 per case; MAI-DxO cut roughly $3,000 of that cost but added only about four points of accuracy. With GPT-5.5, released in April 2026, accuracy on complex cases rose from 72 percent to nearly 87 percent at slightly lower cost. On emergency cases, GPT-5.5 alone and with MAI-DxO both scored about 85 percent at similar cost. For Gemini 2.5 Flash, accuracy rose from 46.4 percent alone to nearly 69 percent with MAI-DxO, but the gap between weaker and stronger models was still larger than the orchestrator's boost.

TechCrunch's reading is that most of the improvement comes from the underlying model, with the orchestrator adding on top. Microsoft said gains shrink as models near the theoretical maximum.

Three reasons to hold the superintelligence label

The comparison is lopsided. NEJM cases of this kind are normally handled by specialists or physician teams; TechCrunch notes that neither was in the comparison. Microsoft's page does not state the number of physicians or how they were recruited.

It is a simulation. The cases are written records turned into stepwise puzzles, and the benchmark was designed by the same team that built the system. Microsoft says results may not transfer directly to routine practice.

The definition is generous. Microsoft defines medical superintelligence as "the capacity to surpass the strongest contemporary human-led health systems in medical reasoning." Beating single generalists on a curated benchmark does not meet that definition. For what the term "superhuman" can honestly mean, see our explainer on narrow versus general superhuman intelligence.

What happens next

Microsoft is working with Beth Israel Deaconess Medical Center in Boston on a prospective study in which patients who consent receive standard care and also engage with MAI-DxO, and its recommendations are compared with those of human doctors. Microsoft is also working with the Mayo Clinic on a healthcare foundation model planned for 2027. TechCrunch reports Dominic King, Microsoft AI's vice president of health, said "There is something very substantial and exciting here," and Suleyman expects health models to improve by "an order of magnitude" in capability within 24 to 36 months.

What a fair test would look like

A convincing next step would add physician teams as a comparison, report the number and recruitment of the physicians, and publish all test cases so others could rerun them. It would also report outcomes, not only diagnoses: whether following the system's recommendation leads to better treatment decisions and fewer unnecessary tests in a real hospital. The Beth Israel prospective study is designed to compare the system's recommendations against human doctors' independent assessments, which is closer to that standard than any simulation.

Until those checks exist, the safest summary is narrow: on a well-designed benchmark, a panel of AI agents did better than individual doctors, and the margin depended mostly on which base model sat underneath. That is a useful engineering result and a reasonable reason to run the clinical study. It is not yet a reason to change how any hospital diagnoses patients.

Our take

This is a good paper about a useful technique: have models argue before they order tests, and count the cost. It is not yet evidence that the system improves outcomes. We would judge it by the Beth Israel study, by whether physician teams are added as a comparison, and by whether the results hold on cases the models could not have seen.