On-premise medical AI agent matches cloud diagnostic accuracy

A locally hosted clinical AI agent built on the open-weight Qwen-3.5 model reached 90.04% diagnostic accuracy on a seven-disease MIMIC-IV benchmark, trailing a GPT-5.2 cloud baseline by 0.7 percentage points, according to a Nature Medicine paper published September 15. On the harder four-disease CDM task the on-premise system landed at 83.8%.

The interesting part is not the raw score. The agent pairs local execution with a reliability gate that lets clinicians tune autonomy per case. Behavioral consistency across repeated queries turned out to be the strongest correctness signal, with an AUC of 0.860. Set the consistency threshold at 0.90 and the system defers on roughly half of cases — 49.4% coverage — while hitting 98.9% accuracy on what it retains.

The authors argue that "reliability signals can be quantified at decision time and translated into explicit triage policies within a locally governed workflow," and that consistency-based gating "retained a substantially larger autonomous subset than probability-based gating at similar retained-set accuracy."

Stress-testing exposed the ceiling. When the study introduced an ungrounded patient agent that fed the pipeline noisier or contradictory input, accuracy fell from 90.6% to 70.2%, a 20.3-point decline. Physician adjudication of 181 MIRA-v2 cases found 81.8% clinically valid for both the agent and the EHR reference label, with another 4.4% judged valid despite disagreeing with the EHR. An LLM-based evaluator matched the physician consensus 92.3% of the time.

Two researchers on our tracker circulated the paper shortly after it appeared.