Nature Medicine: on-prem clinical AI nears cloud accuracy

A study published in Nature Medicine reports that open-weight language models running fully on-premise can approach cloud diagnostic performance in a hospital setting: Qwen-3.5 reached 90.04% accuracy on the MIRA-v2 benchmark of seven acute conditions, against a 90.7% cloud baseline from GPT-5.2.

The authors, with Jakob Nikolas Kather as corresponding author, argue the harder problem is not raw accuracy but knowing when to trust an answer. They introduce a confidence signal called ConsistencyDx, which measures semantic agreement across repeated runs of the same case rather than reading token-level probabilities. In dual-stream ROC analysis it reached AUC = 0.860, well above the 0.747 that internal probability scores managed, and it held up under perturbation testing with AUC = 0.875 while probability-based metrics stayed overconfident.

The point is triage. "At a consistency threshold of 0.90, 49.4% of cases were retained at 98.9% diagnostic accuracy," the paper reports; the remaining cases route to clinician review. On the CDM benchmark of 2,400 abdominal cases Qwen-3.5 scored 83.8% and GLM-4.5-Air 81.2%, and on the external VivaBench set of 990 cases top-1 accuracy dropped to 72.22%. A subset of 181 MIRA-v2 cases underwent blinded physician adjudication.

The authors are explicit about what safety means here. "Safety, therefore, depends on reliable decision-time signals indicating when to trust, verify or defer," they write. They also flag that both primary benchmarks derive from MIMIC-IV, "reflecting a single-institution data ecology," and that "absolute consistency thresholds are not universal and must be calibrated to the deployment configuration rather than transferred directly across settings."