TU Dresden on-premise AI hits 90% on MIMIC-IV diagnosis test

An on-premise clinical AI agent from TU Dresden, reported in Nature Medicine on September 15, 2026, reached 90.04% accuracy on a seven-disease diagnostic task drawn from MIMIC-IV, close to a GPT-5.2 cloud baseline at 90.7%.

The paper, with Li Zhang as first author and Prof. Jakob N. Kather leading the group, puts four open-weight models through the primary benchmark running locally: Qwen-3.5 at 90.0%, GLM-5 at 89.7%, GLM-4.5-Air at 88.4%, and GPT-OSS at 85.3%. On a second benchmark of 2,400 abdominal cases (CDM), Qwen-3.5 hit 83.8%. Two researchers we track shared the paper the same week it landed.

The more interesting claim sits in a reliability framework. The authors evaluated three confidence signals (token probability, linguistic hedging, and behavioral consistency across independent runs) and found the last one was the strongest predictor of correctness. "Diagnostic behavioral consistency provided the strongest discrimination of correctness (AUC = 0.860)," the paper reports. At a consistency threshold of 0.90, the agent retained 49.4% of cases for autonomous handling at 98.9% accuracy, routing the remainder to clinician review.

A stress test cut into the confidence story. When the simulated patient agent was starved of grounding information, overall accuracy fell from 90.6% to 70.2%, and internal-probability measures became overconfident even as consistency-based gating stayed discriminative (AUC = 0.875).

The authors flag limits: benchmarks derived from a single institution, text-only inputs with no native image interpretation, retrospective simulations only, and lower accuracy in older age groups requiring dedicated bias auditing. Five-run consistency analysis also multiplies token use roughly five-fold.