Nature Medicine: on-prem clinical AI hits 90% accuracy

A fully on-premise clinical AI agent hit 90.04% diagnostic accuracy on a seven-disease MIMIC-IV benchmark, effectively matching a GPT-5.2 cloud baseline of 90.7%, according to a paper published in Nature Medicine.

The point of the paper is less the headline accuracy than the confidence routing. The authors frame the clinical-translation gap as "two unmet requirements: institutionally governed deployment and reliable decision-time uncertainty estimation," and their answer is a reliability layer that measures how consistently the agent lands on the same diagnosis across repeat runs. That behavioral consistency turned out to be the strongest correctness signal in the study, with an AUC of 0.860, well above the model's internal probability score at 0.747.

The practical consequence: at a consistency threshold of 0.90, the system kept 49.4% of cases for autonomous handling and got 98.9% of them right. Everything below that threshold routes back to clinicians. Physician adjudication of 181 cases found 81.8% clinically valid for both the agent and the reference diagnosis, with agreement between an LLM evaluator and the physician panel running at 92.3%.

Open-weight models Qwen-3.5, GLM-5, GLM-4.5-Air and GPT-OSS were all tested; the top on-prem configuration landed within a point of the cloud baseline. On a separate four-disease abdominal benchmark, Qwen-3.5 reached 83.8%, against a prior open-weight best of 70.5%.

The authors are careful about the ceiling. "Confidence-based routing is a mechanism for risk stratification, not a substitute for continuing clinical governance," they write.