On-premise Qwen-3.5 matches cloud on Nature Medicine test

Qwen-3.5 running entirely on hospital hardware scored 90.0% on a seven-condition diagnostic benchmark in Nature Medicine, within 0.7 points of GPT-5.2 running in the cloud at 90.7%.

The paper, led by Jakob Nikolas Kather, tests four open-weight models against GPT-5.2 as the cloud baseline: Qwen-3.5 at 90.0%, GLM-5 at 89.7%, GLM-4.5-Air at 88.4%, and GPT-OSS at 85.3%. On CDM, a 2,400-case set of four abdominal diagnoses, Qwen-3.5 posts 83.8%. The previous best open-weight score on that benchmark was 70.5%.

Raw accuracy is only half the pitch. The team's framework tries to say when the agent should act on its own and when it should defer. "Diagnostic behavioral consistency provided the strongest discrimination of correctness (area under the curve (AUC) = 0.860)," the paper reports. At a consistency threshold of 0.90, the agent retained 49.4% of cases at 98.9% accuracy, with three residual errors among 272 retained cases. The rest are meant to be deferred to a clinician.

The authors split the trust question in two: "operational trust" over the governance of data and models, and "decisional trust" over decision-time reliability estimation. Two researchers we follow circulated the paper the day it went up.

No prospective clinical use is reported, and the paper does not publish per-hardware runtime figures for the repeated-sampling consistency check.