On-prem Qwen-3.5 matches cloud GPT-5.2 on Nature Med tests

Open-weight medical AI running entirely on hospital hardware nearly matched a leading cloud model on a seven-disease diagnostic benchmark, in a paper published this week in Nature Medicine by a team led by Jakob Nikolas Kather.

The on-premise agent, built on Qwen-3.5, scored 90.0% on the MIRA-v2 benchmark of 551 cases across seven conditions. Cloud baseline GPT-5.2 scored 90.7%. On a separate set of 2,400 cases covering four abdominal conditions, Qwen-3.5 reached 83.8%, with GLM-4.5-Air close behind at 81.2%. Physician adjudication validated 81.8% of reviewed cases as clinically valid for both the agent and the reference diagnosis.

The more novel piece is a decision-time reliability layer. Asking the agent to reason through each case multiple times and measuring whether it lands on the same diagnosis — what the paper calls behavioral consistency — outperformed the model's internal likelihood as a signal of correctness, with an AUC of 0.860 against 0.747. At a consistency threshold of 0.90, the system retained 49.4% of cases and got 98.9% of those right; the rest are flagged for a clinician.

Under induced information scarcity, accuracy fell from 90.6% to 70.2%, though the consistency signal stayed discriminative (AUC 0.875). "Safety, therefore, depends on reliable decision-time signals indicating when to trust, verify or defer," the authors write.

Two of the clinicians in our Who's Who tracker had shared the paper by the day after it posted.