On-premise clinical AI hits 90% accuracy in Nature Medicine

A fully on-premise clinical AI agent from Jakob Nikolas Kather's group hit 90.04% accuracy on a seven-disease diagnostic benchmark and 83.8% on a four-condition set, according to a Nature Medicine paper published September 15.

The system runs entirely locally, and its authors put the emphasis on knowing when to defer. They call the approach 'selective autonomy': a behavioral-consistency score decides which cases the agent handles on its own and which get sent to a clinician.

The most discriminating reliability signal was behavioral consistency, with an AUC of 0.860, ahead of linguistic hedging in the reasoning trace (0.792) or token-probability scoring (0.747). At a consistency threshold of 0.90, the agent retained 49.4% of the 551 MIRA-v2 cases for autonomous handling and reached 98.9% accuracy on that retained slice, with only three residual errors.

'Behavioral consistency was the strongest signal, not because it eliminates uncertainty but because it helps determine when autonomous handling may be reasonable,' the authors write.

On the open-weight side, Qwen-3.5 came within a point of the GPT-5.2 cloud baseline on MIRA-v2 (90.0% versus 90.7%) and posted 83.8% on the CDM benchmark, up from a prior open-weight best of 70.5%. Physician adjudication of 181 cases found 81.8% of both agent and reference diagnoses clinically valid.

Under stress testing with withheld patient history, consistency scores 'decreased sharply in parallel with accuracy' while probability scores 'remained high despite lower accuracy,' evidence the authors offer for treating behavioral stability as a better trust signal than raw token likelihood in a clinical setting.