On-prem medical AI agent hits 90% on MIMIC-IV diagnostic task

An on-premise clinical AI agent reached 90.04% accuracy on a seven-disease diagnostic task drawn from MIMIC-IV, approaching a cloud baseline of 90.7% (GPT-5.2), according to a paper by Li Zhang, Jakob Nikolas Kather and colleagues published in Nature Medicine. On a harder four-disease task (CDM), the same system reached 83.8%.

The point of the paper is not raw accuracy. It is what the authors call selective autonomy: the agent produces multiple runs per case and scores their behavioral consistency, and that signal separated correct from incorrect diagnoses with an area under the curve of 0.860, holding up under stress testing at 0.875. "At a consistency threshold of 0.90, 49.4% of cases were retained at 98.9% diagnostic accuracy," the paper reports, with three autonomous errors across 272 retained cases. The remaining half get deferred to a clinician.

The authors frame the design around what they call operational trust, defined as "governance of data and models, including privacy, auditability, version stability and deployment control." They tested Qwen-3.5, GLM-5, GLM-4.5-Air, GPT-OSS, Gemma-3 and MedGemma running locally, with Qwen-3.5 leading the on-prem field. Two of the AI researchers we track had shared the paper by the time it landed in our feed.

The robustness picture is mixed. Under perturbation stress testing, baseline accuracy of 90.6% dropped to 70.2%, a 20.3 percentage point fall. Physician adjudication of 181 MIRA-v2 cases judged both diagnoses clinically valid in 81.8% of cases, and found the agent's diagnosis valid despite disagreeing with the EHR label in a further 4.4%. The five-perspective consistency check costs roughly five times the tokens of a single inference pass.