On-prem medical AI hits 90% on MIMIC-IV, defers uncertain cases

The Nature Medicine paper reports 90.04% accuracy on a seven-disease task drawn from MIMIC-IV. But the more interesting number is 98.9%. That is the diagnostic accuracy on the 49.4% of cases the agent chose to keep under a behavioral-consistency gate; the other 279 cases were deferred. Three autonomous errors in 272 retained cases.

The setup, from Li Zhang, Georg Wölflein, Dyke Ferber, Jakob Nikolas Kather and colleagues, runs on-premise rather than through a cloud API. A Physician Agent and Patient Agent operate locally, running multi-turn dialogue with tool use for physical exam, labs, imaging and microbiology. Each case is run five times; the agreement across those runs is the reliability signal that decides whether the model answers or hands off.

Behavioral consistency out-performed token-level confidence. ProbScore came in at AUC 0.747; linguistic certainty at 0.792. "ConsistencyDx provided the strongest discrimination of diagnostic correctness," the paper reports, at AUC 0.860, and the signal held up when accuracy collapsed from 90.6% to 70.2% under perturbed patient information, staying at AUC 0.875.

The paper is explicit about cost. Running five stochastic passes per case increases token use "approximately five-fold versus single-pass inference." It adds: "Absolute consistency thresholds require calibration to deployment configuration," meaning a hospital cannot just copy the 0.90 threshold across sites. Age-stratified analysis revealed lower accuracy in older patients, which the authors flag as needing dedicated bias auditing.

For reference, a GPT-5.2 cloud baseline reached 90.7% on the same seven-disease task; on-prem Qwen-3.5 hit 90.0% and GLM-5 hit 89.7%. Two of the AI experts we track posted the paper on our radar.