On-premise clinical AI agent hits 90% on MIRA-v2 benchmark

A Nature Medicine paper from a team led by Li Zhang, Georg Wölflein and Dyke Ferber reports an on-premise clinical AI agent reached 90.04% accuracy on the MIRA-v2 seven-disease benchmark, within 0.7 percentage points of a GPT-5.2 cloud baseline. On the harder CDM four-disease task, the same Qwen-3.5-based stack scored 83.8%, well ahead of the 70.5% posted by the previous best open-weight model.

The point of the paper is not the raw score but the reliability layer bolted onto it. The authors evaluated four decision-time confidence signals; behavioral consistency across five repeated runs was the strongest discriminator of correctness, with AUC of 0.860. At a consistency threshold of 0.90, the agent kept 49.4% of cases and hit 98.9% accuracy on those retained, deferring the other 279 cases to clinician review.

"Harm depends not only on average performance but also on whether an agent handles uncertainty appropriately, routing unstable or unreliable outputs toward human oversight rather than presenting them as definitive decisions," the paper states. Under a stress test that stripped patient context, overall accuracy collapsed from 90.6% to 70.2%, but the consistency metric still separated right from wrong (AUC 0.875) while the internal-probability and linguistic-certainty signals miscalibrated.

The catch: both primary benchmarks draw from MIMIC-IV, a single institution; the system reasons over text only, with no native image interpretation; and every evaluation is a retrospective simulation. Computing five parallel runs raises token cost roughly five-fold over a single-pass call. Physician adjudication of 181 MIRA-v2 cases judged 81.8% of agent diagnoses clinically valid.

Two of the researchers we follow in our Who's Who directory had already circulated the paper. The authors' own framing stays measured: "governance-oriented local deployment did not impose a large performance penalty in this setting."