Qwen-3.5 on-prem agent matches GPT-5.2 on diagnostic test

An on-premise clinical AI agent running open-weight models scored 90.04% accuracy on a seven-disease diagnostic task drawn from MIMIC-IV records, matching a GPT-5.2 cloud baseline that reached 90.7%. That is the headline result from a paper published in Nature Medicine on September 15, 2026.

Four open-weight models were run through the same pipeline. Qwen-3.5 landed at 90.0%, GLM-5 at 89.7%, GLM-4.5-Air at 88.4%, and GPT-OSS at 85.3%. The gap to the closed cloud system on this benchmark is now under a percentage point.

The more interesting move is what the authors put around the raw accuracy. They rerun each case five times and check whether the diagnoses agree, calling the signal behavioral consistency. On the 551-case MIRA-v2 set, the paper reports that "diagnostic behavioral consistency provided the strongest discrimination of correctness (area under the curve (AUC) = 0.860)." Set the consistency threshold at 0.90 and, per the paper, "49.4% of cases were retained at 98.9% diagnostic accuracy," with three residual errors. The remaining half is deferred to a clinician.

The stress test is the part that keeps this honest. When patient testimony was perturbed to induce information scarcity, accuracy fell from 90.6% to 70.2%. The consistency AUC held at 0.875. The system got more wrong, and still recognized when it was uncertain.

The abstract itself frames the claim narrowly: "Autonomous clinical artificial intelligence (AI) agents powered by large language models (LLMs), meaning systems that can complete a diagnostic workflow without continuous human input, are increasingly capable of supporting complex reasoning and decision-making." No live-hospital deployment is claimed. Two of the medical-AI researchers we follow posted the link the same day the paper appeared.