On-prem clinical AI agent hits 90% on MIMIC-IV diagnosis test

An on-premise clinical agent built on open-weight models reached 90.04% accuracy on a seven-disease diagnostic benchmark drawn from MIMIC-IV ICU data, roughly matching a cloud baseline running GPT-5.2 at 90.7%, according to a Nature Medicine paper published on 15 September 2026.

The study, led by researchers including Jakob Nikolas Kather, tests four open-weight models (Qwen-3.5, GLM-5, GLM-4.5-Air, and GPT-OSS) inside what the authors call a multi-perspective reliability framework designed to let the agent hand borderline cases back to a clinician. On the seven-condition MIRA-v2 task, Qwen-3.5 topped the on-prem group at 90.0%, with GLM-5 at 89.7%, GLM-4.5-Air at 88.4%, and GPT-OSS at 85.3%. On a separate four-disease CDM benchmark, Qwen-3.5 reached 83.8%.

The reliability signal the authors lean on is behavioral consistency across repeat prompts. It achieved an AUC of 0.860 for separating correct from incorrect diagnoses, better than the models' internal probability scores at 0.747. At a consistency threshold of 0.90, the agent kept 49.4% of cases and hit 98.9% accuracy on that retained slice. The rest would in practice be routed to a doctor.

"Behavioral consistency was the strongest signal, not because it eliminates uncertainty but because it helps determine when autonomous handling may be reasonable and when uncertainty should remain with the clinician," the authors write.

Physician adjudication of a sample of disagreements found 81.8% of reviewed cases clinically valid for both the agent and the EHR reference diagnosis, and in 4.4% of cases the agent's diagnosis was judged valid even when it disagreed with the EHR record. The paper also reports a Gemini 3.1 Flash-Lite evaluator agreed with the physician consensus 92.3% of the time (Gwet's AC1 = 0.898). Two researchers we follow posted the paper the day it went live.