On-prem clinical AI agent hits 90% accuracy on MIMIC-IV tasks

An on-premise clinical AI agent hit 90.04% diagnostic accuracy on a seven-disease task and 83.8% on a four-disease task, both derived from MIMIC-IV, while running entirely inside a hospital's own infrastructure. The work, published in Nature Medicine, pairs the agent with a reliability layer that decides, per case, whether the model should act on its own or hand off to a clinician.

The signal doing the work is behavioral consistency: how stably the model returns the same diagnosis under repeated queries. "Diagnostic behavioral consistency provided the strongest discrimination of correctness (area under the curve (AUC) = 0.860)," the paper reports, and that signal held up under stress testing at AUC 0.875. Set the consistency threshold at 0.90 and 49.4% of cases pass through at 98.9% accuracy; the rest are routed for review.

Open-weight models carried the benchmarks. Qwen-3.5 topped the seven-disease task, with a cloud comparator, GPT-5.2, at 90.7%. The authors describe this as "approaching a cloud baseline on the primary benchmark" rather than matching it. An older open-weight system, Gemma-3, reached 70.5% on the four-disease task. On physician review, the authors report 81.8% of reviewed cases were clinically valid for both the agent diagnosis and the reference label.

Two of the AI researchers we follow shared the paper's link within days of publication.