On-premise clinical AI agent hits 90% in Nature Medicine test

A fully on-premise clinical AI agent hit 90.04% accuracy on a seven-disease diagnostic task and 83.8% on a four-disease abdominal benchmark. The more interesting number is what happens when you let it decline to answer.

In a paper in Nature Medicine, researchers describe a clinical agent that runs entirely on local hardware, using open-weight models such as Qwen-3.5, GLM-5, GLM-4.5-Air and GPT-OSS instead of a cloud API, wrapped in what the authors call a multi-perspective reliability framework for estimating confidence at decision time. They compared three signals: a probabilistic score (AUC = 0.747), linguistic certainty inside the reasoning trace (AUC = 0.792), and diagnostic behavioral consistency across repeated runs, which came out strongest at AUC = 0.860.

The trade-off is stated plainly. "At a consistency threshold of 0.90, 49.4% of cases were retained at 98.9% diagnostic accuracy," the paper reports, with the deferred stream concentrating 49 of 52 total errors for human review. The framework, the authors write, shows that "reliability signals can be quantified at decision time and translated into explicit triage policies within a locally governed workflow."

The catches sit inside the paper itself. Primary benchmarks come from a single institution's data ecology. The agent reasons only over text, with no imaging in the loop. And the five-run consistency estimation costs roughly five times a single-pass inference. On VivaBench, an external physician-curated set of 990 PubMed cases across ten specialty groups, top-1 accuracy fell to 72.22%.

Two of the researchers in our tracked directory had already shared the paper's link by the time we picked it up.