On-premise medical AI matches cloud accuracy in Nature study

On a benchmark of 551 emergency-room cases, an open-weight medical AI running entirely on local hardware diagnosed seven common conditions at 90.04% accuracy, within a percentage point of GPT-5.2's cloud-hosted 90.7%, according to a paper by Li Zhang, Jakob Nikolas Kather and colleagues in Nature Medicine.

The more interesting number is 0.860. That is the AUC the authors report for what they call "behavioral consistency", meaning they run the same case through the model repeatedly and check whether the diagnostic answer stays stable across runs. The signal beat token-level probability (AUC 0.747) and the model's own linguistic hedging in its written reasoning (AUC 0.792), and it held up under adversarial stress testing (AUC 0.875).

That signal drives a triage mechanism. At a consistency threshold of 0.90, the system retained 49.4% of cases (272 of 551) for autonomous handling and referred the rest for physician review; on the retained cases, accuracy was 98.9%, with three residual errors. Among on-premise open-weight models tested, Qwen-3.5 led at 90.0%, followed by GLM-5 at 89.7%, GLM-4.5-Air at 88.4% and GPT-OSS at 85.3%.

Two facts sit alongside the headline. Repeated-run consistency estimation "increased computational token use approximately five-fold relative to single-pass inference," the paper reports. And subgroup analysis found "diagnostic accuracy was lower in older age groups compared to younger adults" on both benchmarks, which the authors flag as requiring dedicated bias auditing before deployment.