Nature Medicine: on-prem clinical AI hits 90% on MIMIC-IV

A fully on-premise clinical AI agent reached 90.04% diagnostic accuracy on a seven-disease MIMIC-IV benchmark and 83.8% on a four-disease benchmark, within 0.7 percentage points of a GPT-5.2 cloud baseline, in a Nature Medicine paper from Jakob Nikolas Kather's group published September 15, 2026.

The team, led by first author Li Zhang, wrapped a local model in a three-part reliability layer: token-level probability, linguistic hedging in the response, and behavioral consistency across five separate runs. "Behavioral consistency was the strongest signal," the authors write, "not because it eliminates uncertainty but because it helps determine when autonomous handling may be reasonable." That signal returned an AUC of 0.860 for discriminating correct from incorrect diagnoses, and 0.875 under stress testing with degraded evidence.

The selective-autonomy number is the one that will get read closely. At a consistency threshold of 0.90, the system retained 272 of 551 MIRA-v2 cases for autonomous handling with only 3 residual errors, concentrating the remaining 49 errors in the review stream. That is 49.4% coverage at 98.9% accuracy. Physician adjudication of a subset found 81.8% of reviewed cases clinically valid for both agent and reference label, and 4.4% valid only for the agent's diagnosis. Two of the researchers on our Who's Who list circulated the source link this week.

Costs and caveats sit inside the paper itself. Five-run consistency estimation multiplies token use roughly five-fold over single-pass inference. Diagnostic accuracy was lower in older age groups than in younger adults across both benchmarks. The authors position the work as evidence for a governance model rather than a replacement one: "These findings support a practical framework for institutionally governed clinical agents in which decision-time reliability signals identify a lower-risk subset for autonomous handling and defer the remainder for review."