On-premise medical AI matches GPT-5.2 on Nature benchmark

A fully on-premise medical AI agent reached 90.04% diagnostic accuracy on a seven-condition benchmark, close to the 90.7% posted by a GPT-5.2 cloud baseline, according to a paper published 15 September 2026 in Nature Medicine.

The system, from Jakob Nikolas Kather's group with Li Zhang as lead author, runs entirely inside a hospital's walls. It pairs a Physician Agent with a Patient Agent in multi-turn dialogue, then scores each candidate diagnosis on three confidence signals: token-level likelihood, linguistic hedging, and cross-run behavioral consistency across five stochastic runs.

The behavioral signal did the heavy lifting. "Behavioral consistency was the strongest signal, whereas language-derived metrics contributed complementary information, particularly in the reasoning stream," the authors write. On the MIRA-v2 benchmark (n=551), consistency alone gave an AUC of 0.860 for separating correct from incorrect diagnoses.

Set the consistency threshold at 0.90 and the agent runs autonomously on 49.4% of cases at 98.9% accuracy, with three residual errors. The other half is deferred to a clinician.

The local models were named. Qwen-3.5 hit 90.0% on MIRA-v2, GLM-5 hit 89.7%, GLM-4.5-Air 88.4%, and GPT-OSS 85.3%. Cloud GPT-5.2 hit 90.7%.

The system did not hold up under stress. When patient testimony was perturbed to induce information scarcity, accuracy fell from 90.6% to 70.2%, though consistency still discriminated correctness at AUC 0.875, meaning the model got more wrong but still recognised when it was uncertain.

In blinded physician review of 181 cases, both the agent's and the EHR's diagnosis were judged clinically valid 81.8% of the time; the agent's call was judged valid despite disagreeing with the EHR in a further 4.4%. The paper separates "operational trust" (governance, privacy, auditability) from "decisional trust", the reliability signals a clinician needs to know when to rely on the agent.