On-prem medical AI agent hits 90% accuracy in Nature paper

A fully on-premise clinical AI agent reached 90.04% diagnostic accuracy on a seven-disease benchmark, nearly matching a cloud baseline built on GPT-5.2 at 90.7%, according to a paper published in Nature Medicine.

The system, described by Li Zhang, Jakob Nikolas Kather and colleagues, runs open-weight models locally, including Qwen-3.5, GLM-4.5-Air, GLM-5, GPT-OSS and MedGemma, so hospitals do not have to route patient data through an external API. On the CDM benchmark of 2,400 abdominal cases, Qwen-3.5 reached 83.8% accuracy against 70.5% for the previous best open-weight result, Gemma-3.

The authors emphasise something other than raw accuracy: knowing when to trust the agent. "Clinical translation depends not only on diagnostic capability but also on safe institutional governance and appropriate clinician reliance," they write, arguing that "safety, therefore, depends on reliable decision-time signals indicating when to trust, verify or defer."

Their strongest signal was diagnostic behavioral consistency, which achieved an AUC of 0.860 for discriminating correct from incorrect answers, well above token-probability scoring (0.747) or the model's own verbal certainty (0.792). At a consistency threshold of 0.90, the agent retained 49.4% of cases at 98.9% accuracy, pushing the rest to human review.

Under stress testing with an ungrounded Patient Agent that fed unreliable histories, overall accuracy dropped from 90.6% to 70.2%, but the consistency-based confidence metric held at AUC 0.875. In a physician adjudication of 181 disagreement cases, 81.8% of labels were valid for both the agent and the electronic health record, and 4.4% were valid for the agent only.

The authors are candid about the trade: "selective autonomy does not remove uncertainty but redistributes it."