On-premise Qwen-3.5 matches cloud GPT-5.2 on ED diagnosis

An open-weight language model running entirely inside a hospital's own compute can match a cloud frontier system on emergency-room diagnostic classification, according to research published in Nature Medicine by a team led by Jakob Nikolas Kather. On the MIRA-v2 benchmark of 551 MIMIC-IV cases spanning seven acute conditions, Qwen-3.5 reached 90.0% accuracy, within 0.7 percentage points of GPT-5.2 at 90.7%. On the larger CDM benchmark of 2,400 four-disease cases, the same on-premise model scored 83.8%.

The framing the paper pushes is that raw accuracy is the wrong headline. What the authors build around it is a multi-perspective reliability signal that flags when the model's own answer should not be trusted. Cross-run behavioral consistency, running the same case multiple times and measuring how stable the diagnosis is, separated correct from incorrect answers at AUC 0.860, beating a token-probability score at 0.747. Setting a 0.90 consistency threshold retained 49.4% of cases for autonomous handling at 98.9% accuracy, with only 3 residual errors among 272 retained cases.

"Behavioral consistency was the most informative signal in this setting, not because it eliminates uncertainty but because it helps determine when autonomous handling may be reasonable and when uncertainty should remain with the clinician," the paper argues. An adversarial test in which the Patient Agent was ungrounded dropped overall accuracy from 90.6% to 70.2%, a reminder that these numbers assume the intake side works.

The five-run consistency estimation costs roughly five times the tokens of a single-pass answer, a tradeoff the authors flag directly.