Nature Medicine: on-prem AI agents match cloud on diagnoses

On a seven-disease diagnostic benchmark of 551 cases, an open-weight model running inside a hospital came within 0.7 percentage points of a frontier cloud model. According to the paper in Nature Medicine, Qwen-3.5 hit 90.0% accuracy on the MIRA-v2 benchmark; GPT-5.2 in the cloud hit 90.7%. GLM-5 was close behind at 89.7%.

The authors, led by Jakob Nikolas Kather, are direct about why the on-prem number matters. "Clinical translation, however, remains limited by two unmet requirements: institutionally governed deployment and reliable decision-time uncertainty estimation," the abstract reads. Elsewhere they add: "Operational trust concerns governance of data and models, including privacy, auditability, version stability and deployment control, motivating on-premise or similarly governed deployment in healthcare."

To handle the second requirement, the paper introduces a behavioral-consistency signal, ConsistencyDx, which the authors describe as "the strongest indicator of diagnostic correctness" (AUC = 0.860, holding at 0.875 under a perturbation stress test). Applied as a gate, it trades coverage for reliability. "At a consistency threshold of 0.90, 49.4% of cases were retained at 98.9% diagnostic accuracy," the paper reports. The other half of cases get deferred.

The evaluation spans three benchmarks (MIRA-v2 with 551 cases across seven conditions, CDM with 2,400 cases across four abdominal diagnoses where Qwen-3.5 scored 83.8%, and VivaBench with 990 cases across ten specialty groups) and pits four on-premise models (GLM-4.5-Air, GLM-5, Qwen-3.5, GPT-OSS) against GPT-5.2 as the cloud baseline.

The privacy claim is stated in one sentence: "No protected health information or deidentified clinical text was transmitted to, stored by or accessible to any external entities." Two researchers we track circulated the paper.