Microsoft's MAI-DxO Beats Doctors 83% to 36% on NEJM Cases
TL;DR
- MAI-DxO on GPT-5 scored 83.2% on 304 NEJM teaching cases versus 36.1% for generalists with internet access, a 2.3x gap on text-only historical records.
- The physician comparison excluded specialists and multidisciplinary teams, the standard of care in real hospital diagnostics, giving critics a documented methodological objection.
- AI cost-per-case estimates ($2,241 to $5,431) undercut physician workup costs ($3,049 to $4,725), but the paper cannot establish actual hospital savings without workflow and false-alarm data.
Microsoft's MAI-DxO, an orchestration layer that runs AI agents as a mock physician panel, correctly diagnosed 83% of complex case reports from the New England Journal of Medicine. The generalist doctors and internists benchmarked against it, working with reference materials and without a specialist team, got 36%. The paper was published Thursday in Science. TechCrunch reports that Microsoft tested the harness against frontier models from OpenAI, Google, Anthropic, Meta, and DeepSeek, with the strongest accuracy coming paired with OpenAI's GPT-5 and GPT-5.5.
Microsoft AI CEO Mustafa Suleyman is pitching the work as a step toward "medical superintelligence," which Microsoft defines as "the capacity to surpass the strongest contemporary human-led health systems in medical reasoning and coordination across the full arc of care." Dominic King, VP of Health at Microsoft AI, put the use case more plainly: "The biggest challenge in healthcare delivery at the moment is the growing mismatch between demand and supply of medical expertise."
Most of the lift sits in the underlying model, not the harness. DxO moved Gemini 2.5 Flash from 46.4% to nearly 69%, pushed GPT-5.5 from 72% to nearly 87%, and on GPT-5 trimmed roughly $3,000 off an approximately $8,400 per-case cost while adding only about four accuracy points. TechCrunch notes that "the study's own data suggests that much of the improvement comes from more powerful underlying AI models." The NEJM cases being graded are also usually handled by specialists or physician teams, who were not part of the comparison.
Microsoft is partnering with Beth Israel Deaconess Medical Center in Boston to evaluate DxO's clinical reasoning in hospital settings, and plans a healthcare foundation model with Mayo Clinic targeted for 2027. "I really believe that in 2027, we will be on the verge of a coding style breakthrough for medical records," Suleyman said. It joins a steady run of Microsoft AI stories we have tracked this quarter.
What others are reporting
-
Unite.AI Read →
Details the Gatekeeper and Judge agent architecture, publishes cost-per-case figures ($2,241-$5,431 vs $3,049-$4,725), and covers the Mayo Clinic partnership and Beth Israel prospective trial.
83.2% accuracy for the system on New England Journal of Medicine case records against 36.1% for physicians using reference resources.
-
Cornford and Cross Read →
Systematic methodological deconstruction: generalists tested solo, text-only historical cases, no false-alarm data on common conditions, all authors Microsoft-affiliated.
The benchmark's cost estimates cannot by themselves show that a hospital would spend less.
-
WebProNews Read →
Leads with the Beth Israel Deaconess prospective trial as the primary news hook, framing the Science paper as a bridge to real clinical deployment rather than a standalone benchmark.
-
Remio Read →
Frames MAI-DxO's inspectable intermediate diagnostic choices as the real methodological contribution and sets independent reproduction plus patient outcomes as the bar for clinical claims.
Choosing the next investigation creates different opportunities for failure.
-
Crypto Briefing Read →
Emphasizes the open-source SDBench release and near-term hospital deployment pathway, positioning MAI-DxO as a clinical tool with independent validation infrastructure already in place.
MAI-DxO reached accuracy of approximately 83%. Generalist physicians scored between 22% and 36%.
-
News Directory 3 Read →
The only piece to directly quote Suleyman's 'medical superintelligence' language alongside the specialist-exclusion caveat, exposing the gap between the marketing frame and the study scope.
an important step toward medical superintelligence
Originally reported by techcrunch.com
Read the original article →Original headline: Microsoft's MAI-DxO Diagnoses 83% of NEJM Cases vs. 36% for Doctors in Science Paper