On-prem medical AI agent hits 90% accuracy in Nature study

A fully on-premise clinical AI agent hit 90.04% diagnostic accuracy on a seven-disease task and 83.8% on a four-disease task, according to a study published in Nature Medicine on September 15. The best open on-prem model, Qwen-3.5, came within 0.7 percentage points of the cloud baseline GPT-5.2, which reached 90.7% on the same benchmark.

The number that does more work is buried in the reliability layer. At a consistency threshold of 0.90 across repeated runs, "49.4% of cases were retained at 98.9% diagnostic accuracy." The remaining cases get deferred to clinicians. The framework is trying to answer a question hospitals actually care about: when should the model be trusted to answer, and when should it hand the case back.

The team, whose senior author is Jakob Nikolas Kather, tested Qwen-3.5, GLM-5, GLM-4.5-Air and GPT-OSS on three benchmarks: MIRA-v2 (n=551, seven conditions), CDM (n=2,400, four abdominal conditions) and the external VivaBench (n=990). Behavioral consistency across repeated runs (AUC = 0.860) proved a stronger correctness signal than probabilistic confidence. Under induced information scarcity, consistency "decreased sharply in parallel with accuracy" while other scores stayed artificially elevated.

The subgroup finding is stated flatly: "diagnostic accuracy was lower in older age groups than in younger adults," which the authors flag as requiring "dedicated bias auditing" before deployment. Two researchers we follow shared the paper the day it went live.