On-premise medical AI nears cloud model in Nature Medicine
TL;DR
- A fully on-premise clinical agent reached 90.04% accuracy on a seven-disease MIMIC-IV task and 83.8% on a four-disease abdominal task.
- Behavioral consistency across repeated runs was the strongest signal of diagnostic correctness at AUC = 0.860, beating internal-likelihood measures.
- At a 0.90 consistency threshold the agent auto-handled 49.4% of cases at 98.9% accuracy and deferred the rest to clinicians.
The paper's central number: a fully on-premise clinical AI agent hit 90.04% accuracy on a seven-disease task and 83.8% on a four-disease abdominal task, both built from MIMIC-IV records, approaching a cloud baseline on the primary benchmark.
The design pairs local open-weight models with a reliability check. Instead of trusting a single pass, the agent tests whether the diagnosis stays stable across repeated runs. That behavioral-consistency signal, the authors write, 'provided the strongest discrimination of diagnostic correctness in dual-stream receiver operating characteristic (ROC) analysis (AUC = 0.860),' outperforming internal-likelihood and language-based measures and holding up under stress testing.
Applied as a gate, it splits the workload. 'At a ConsistencyDx threshold of 0.90 retained 49.4% of cases (n = 272) while achieving 98.9% diagnostic accuracy in the retained subset,' the paper reports. The rest defer for clinician review.
The Nature Medicine paper put four open-weight models through the primary benchmark: Qwen-3.5 at 90.0%, GLM-5 at 89.7%, GLM-4.5-Air at 88.4% and GPT-OSS at 85.3%, against GPT-5.2 as the cloud baseline at 90.7%. Two researchers we follow flagged it the same week.
Shared on Bluesky by 2 AI experts
Originally reported by nature.com
Read the original article →Original headline: On-premise medical AI agents for reliable clinical decision-making - Nature Medicine