On-prem clinical AI agent nears cloud accuracy in Nature study

A team led by senior author Jakob Nikolas Kather built a clinical AI agent that runs entirely inside a hospital's own infrastructure and reports in Nature Medicine that it hit 90.04% accuracy on a seven-disease diagnostic task drawn from MIMIC-IV. The cloud baseline they compared against, GPT-5.2, scored 90.7% on the same cases.

The gap is small. The more interesting number is what happens when the agent is asked to gate its own answers. The authors quantified internal-likelihood, language-based and behavioral-stability measures across diagnosis and reasoning. "Diagnostic behavioral consistency provided the strongest discrimination of correctness (AUC = 0.860)," they write, and the signal "remained informative under stress testing (AUC = 0.875)."

Set the consistency threshold at 0.90 and, in the paper's words, "49.4% of cases were retained at 98.9% diagnostic accuracy." Three residual errors survived the gate out of 551 cases. Forty-nine errors concentrated in the deferred stream flagged for clinician review.

"Safety, therefore, depends on reliable decision-time signals indicating when to trust, verify or defer," the paper argues. The open-weight models it tested (Qwen-3.5, GLM-5, GLM-4.5-Air and GPT-OSS) were all run in local infrastructure, meaning a hospital would not have to route patient data through a cloud provider to reach near-cloud accuracy. Two researchers in our directory shared the paper the same day it appeared.

On a second MIMIC-IV-derived benchmark of 2,400 cases across four abdominal conditions, the top on-prem model scored 83.8%. The abstract publishes no per-specialty breakdown beyond the seven-disease and four-disease tasks reported.