On-premise open-weight AI agent hits 90% diagnostic accuracy

An on-premise clinical AI agent built on open-weight language models scored 90.04% accuracy on a seven-disease diagnostic benchmark derived from MIMIC-IV, close to a GPT-5.2 cloud baseline that reached 90.7%. The paper appeared in Nature Medicine on 15 September 2026.

The authors, led by Li Zhang with Jakob Nikolas Kather as senior author, tested open-weight models — Qwen-3.5, GLM-5, GLM-4.5-Air, and GPT-OSS — inside a framework designed to run entirely inside a hospital. On a second MIMIC-IV benchmark covering four abdominal conditions, the Qwen-3.5 agent reached 83.8%.

The paper's larger claim is not accuracy alone but a reliability signal that flags when the agent should act on its own and when it should defer. The framework measures three things: internal likelihood, linguistic hedging, and behavioral stability across repeated runs. Behavioral consistency was the strongest predictor of correctness, with an AUC of 0.860 that held at 0.875 under stress testing.

"At a consistency threshold of 0.90, 49.4% of cases were retained at 98.9% diagnostic accuracy," the abstract reports. The rest go back to clinicians.

The benchmarks trace to MIMIC-IV, the ICU dataset from Beth Israel Deaconess Medical Center and MIT. An external check on PubMed-sourced VivaBench cases dropped Qwen-3.5 accuracy to 72.22%, a reminder that the headline numbers sit on one institution's data. The authors frame their contribution as a "practical framework for institutionally governed clinical agents in which decision-time reliability signals identify a lower-risk subset for autonomous handling and defer the remainder for review." Two researchers we track had the paper in circulation the day it published.