On-premise clinical AI agents match GPT-5.2 within 0.7 points

An on-premise clinical AI agent running open-weight models reached 90.04% accuracy on the MIRA-v2 benchmark across seven diseases, landing within 0.7 percentage points of a cloud baseline running GPT-5.2, according to a paper in Nature Medicine.

Parity with the cloud is not the paper's central claim. The team, whose lead authors are Li Zhang, Georg Wölflein and Dyke Ferber, argues clinical translation is blocked by two separate problems: hospitals cannot send patient data to third-party providers, and the agents themselves have no dependable way to signal when they are wrong. Their answer is a multi-perspective reliability framework built around diagnostic behavioral consistency — running the same case through the agent multiple times and measuring whether it converges on the same diagnosis. That signal produced an AUC of 0.860 for identifying incorrect answers, above an internal probability score (0.747) and a measure of the model's linguistic certainty in its reasoning trace (0.792).

"Clinical translation, however, remains limited by two unmet requirements: institutionally governed deployment and reliable decision-time uncertainty estimation," the abstract states.

The practical effect is a triage regime. At a consistency threshold of 0.90, the agent retained 98.9% accuracy on 49.4% of cases — roughly half of encounters marked high-confidence, the rest deferred. Under stress testing with ungrounded patient narratives, overall accuracy dropped from 90.6% to 70.2%, but the consistency signal held at AUC 0.875, meaning the framework still discriminated between trustworthy and untrustworthy answers on the degraded set.

Blinded physician review of 181 cases found 81.8% clinical validity for both agent and reference diagnoses, with 4.4% of agent diagnoses judged valid despite disagreeing with the EHR label. On an external multi-specialty benchmark, VivaBench, the agent hit 72.22% accuracy across 990 cases, and consistency remained the strongest reliability signal (AUC 0.719). The reliability has a price: the five-run consistency estimation increased token usage approximately fivefold compared to single-pass inference.