On-prem clinical AI matches cloud GPT-5.2 in Nature Medicine

An on-premise clinical AI agent hit 90.04% accuracy on a seven-disease diagnostic task drawn from MIMIC-IV, coming within 0.7 percentage points of a cloud GPT-5.2 baseline that scored 90.7%. The finding is reported in Nature Medicine by Li Zhang, senior author Jakob Nikolas Kather and colleagues, who ran open-weight models Qwen-3.5 and GLM-4.5-Air behind the firewall against the cloud baseline. GLM-4.5-Air reached 81.2% on a separate four-condition abdominal benchmark of 2,400 cases.

The paper's more interesting move is what happens once the agent has answered. The authors evaluate three reliability signals: token-level probability, hedging cues in the agent's own language, and cross-run behavioral consistency over five independent stochastic runs. Consistency emerged as the strongest gate on when to trust the model, with an AUC of 0.860 for discriminating correct from incorrect answers, versus 0.747 for the internal probability signal. At a 0.90 consistency threshold, "49.4% of cases were retained at 98.9% diagnostic accuracy," the abstract reports, with the remainder deferred to human review.

Consistency held up under stress. When the authors degraded evidentiary grounding by making the patient-simulation agent less informative, overall accuracy fell from 90.6% to 70.2%, but the consistency signal's AUC rose slightly to 0.875. It "remained the most robust decision-time signal under reduced evidentiary grounding," the paper states. Physician review marked 81.8% of the reviewed cases as clinically valid for both the agent's output and the reference label.

The authors are careful with their own numbers. "Absolute consistency thresholds are not universal," they write, framing the reported cutoffs as calibrated to their deployment rather than portable defaults. Two researchers we track shared the paper the day it went live.