Nature paper: on-prem medical LLMs near cloud accuracy

On MIRA-v2, a 551-case diagnostic benchmark built from MIMIC-IV data, the best on-premise model landed at 90.0% accuracy. GPT-5.2, run in the cloud as the study's baseline, scored 90.7%. The gap was 0.7 percentage points.

That figure sits inside a Nature Medicine paper from a group with Jakob Nikolas Kather as corresponding author. They test five models: GLM-4.5-Air, GLM-5, Qwen-3.5, GPT-OSS and the cloud-hosted GPT-5.2 baseline. On the larger CDM set (2,400 cases across four acute abdominal conditions) Qwen-3.5 hit 83.8% and GLM-4.5-Air 81.2%.

The more interesting move is not the accuracy race. It is how the paper argues an on-prem model should decide when to defer. The authors compare three ways to read a model's confidence: internal token likelihood, the linguistic certainty of its written reasoning, and behavioral consistency across repeated runs. The last one wins. "Behavioral consistency provided the strongest indicator of diagnostic correctness," the paper reports. "Cross-run semantic consistency of the final diagnostic output outperformed token-level likelihood." The AUCs: 0.860 for consistency, 0.792 for linguistic certainty, 0.747 for internal probability.

That gap does real work at the deployment layer. Set the consistency threshold at 0.90 and the system auto-resolves 49.4% of cases at 98.9% accuracy, with only three residual errors from 272 retained. The remaining half route to a clinician.

The on-prem framing is explicit: "Operational trust concerns governance of data and models, including privacy, auditability, version stability and deployment control, motivating on-premise or similarly governed deployment in healthcare." Under a stress test with perturbed inputs, raw accuracy fell from 90.6% to 70.2%, a 20.3-point drop, but the consistency signal held (AUC 0.875). Two researchers on our tracker circulated the paper shortly after it went up.