On-prem clinical AI agent nears GPT-5.2 on Nature diagnosis test
TL;DR
- A fully on-premise clinical agent hit 90.04% accuracy on a seven-disease MIMIC-IV task, versus 90.7% for a GPT-5.2 cloud baseline.
- Diagnostic behavioral consistency across repeated runs was the strongest correctness signal at AUC 0.860, beating internal probability at 0.747.
- At a 0.90 consistency threshold, the agent handled 49.4% of cases autonomously at 98.9% accuracy and deferred the rest to a clinician.
A fully on-premise clinical AI agent scored 90.04% accuracy on a seven-disease diagnostic task drawn from MIMIC-IV records, within a point of a GPT-5.2 cloud baseline at 90.7%. The paper, from Jakob Nikolas Kather's group with Li Zhang as lead author, was published in Nature Medicine on 15 September 2026.
The open-weight models running behind the wrapper were Qwen-3.5, GLM-5, GLM-4.5-Air, and GPT-OSS. On a second, four-disease MIMIC-IV task the same agent scored 83.8%.
The reliability layer is the actual finding. The team wrapped their models in a behavioral-consistency check: run each case multiple times and see whether the diagnoses agree across runs. That signal beat the models' own internal probability scores at flagging correct answers, with an AUC of 0.860 against 0.747, and held at 0.875 under a perturbation stress test.
At a consistency threshold of 0.90, the agent kept 49.4% of cases and hit 98.9% diagnostic accuracy on them. The rest defer to a clinician.
"Behavioral consistency was the strongest signal, not because it eliminates uncertainty but because it helps determine when autonomous handling may be reasonable and when uncertainty should remain with the clinician," the authors write.
Two researchers we follow shared the paper within days of its release.
Shared on Bluesky by 2 AI experts
Originally reported by nature.com
Read the original article →Original headline: On-premise medical AI agents for reliable clinical decision-making - Nature Medicine