TU Dresden's local AI hits 90% on diagnostic benchmark
TL;DR
- An on-premise Qwen-3.5 agent reached 90.0% diagnostic accuracy, within a point of a cloud GPT-5.2 baseline at 90.7% on the same benchmark.
- Behavioral consistency across repeated runs was the strongest predictor of correctness, with an AUC of 0.860 for discriminating right from wrong.
- Using consistency thresholds, 49.4% of cases were retained at 98.9% accuracy, with lower-confidence cases deferred to clinicians.
An AI diagnostic agent running entirely on hospital hardware reached 90.04% accuracy on a seven-disease benchmark, within a point of a cloud-based GPT-5.2 baseline at 90.7%. The study in Nature Medicine, from researchers at TU Dresden's Else Kröner Fresenius Center for Digital Health, tested open-weight models including Qwen-3.5, GLM-5 and GPT-OSS against a shared agent architecture, with the local Qwen-3.5 model at 90.0% coming closest to the cloud baseline.
The team's central move is what they call selective autonomy: the system routes cases it can confidently handle to autonomous processing and defers the rest to clinicians. The strongest signal of correctness turned out to be diagnostic behavioral consistency, the same case producing the same answer across repeated runs, which scored an AUC of 0.860. Using consistency thresholds, 49.4% of cases were retained at 98.9% accuracy.
"Our goal is an AI agent with selective autonomy. These systems should support clinicians in decision-making, but never take over completely," Jakob N. Kather, who led the work, told EurekAlert. First author Li Zhang described the result as "an important step towards more reliable medical AI agents."
On a second benchmark of 2,400 abdominal cases, accuracy was 83.8%. Two researchers we track shared the paper the day it appeared.
Shared on Bluesky by 2 AI experts
Originally reported by nature.com
Read the original article →Original headline: On-premise medical AI agents for reliable clinical decision-making - Nature Medicine