Nature paper: on-prem clinical AI matches GPT-5.2 diagnosis

An on-premise clinical agent running open-weight models scored 90.04% accuracy on a seven-disease diagnostic task drawn from MIMIC-IV records, roughly matching a GPT-5.2 cloud baseline that reached 90.7%, according to a paper published in Nature Medicine on September 15.

The team, with Jakob Nikolas Kather as corresponding author, tested Qwen-3.5, GLM-4.5-Air, GLM-5 and GPT-OSS behind a reliability wrapper the paper calls behavioral consistency: run the same case five times, then see whether the diagnoses agree. That signal, labeled ConsistencyDx, did the real work of separating right answers from wrong ones, with an AUC of 0.860 against 0.747 for a model's own internal probability estimate.

"Diagnostic behavioral consistency provided the strongest discrimination of correctness," the authors write. Set the threshold at 0.90 and the agent retains 49.4% of cases at 98.9% diagnostic accuracy; the rest is deferred to human review through what the paper describes as "confidence-based gating to defer ambiguous or unstable outputs."

Two catches. Five-run consistency estimation increases token use roughly five-fold over single-pass inference. And a multi-agent critic extension built on MedGemma did not lift the numbers, coming in at 87.1% with the critic versus 88.4% without.

On VivaBench, an external validation drawn from ten specialty groups of PubMed case reports, Qwen-3.5 dropped to 72.22% top-1 accuracy on 990 cases. Two of the researchers we track shared the paper the day it landed.