MedGemma-27B rarely notices swapped X-rays if report is present

MedGemma-27B's answer changes on only 4.26 percent of trials when its chest X-ray is swapped for a different image and a radiology report is provided alongside, versus 20.94 percent of trials when no report is present. That is the finding of ModaLens, a paired image-swap audit posted to arXiv.

The authors ran the audit on 3,199 paired MIMIC-CXR cases from 293 patients, testing all 14 questions per case (13 finding-specific and one composite), with each image replaced by one from another study, usually of the same patient. The question and the report stayed fixed. The paired increase in swap sensitivity when the report is removed is 16.7 points, with a patient-clustered 95 percent confidence interval of 15.6 to 17.7. An alternative prompt with a lowercase first-token readout gives a similar gap: 4.70 percent against 17.07 percent.

Written into the abstract itself is a limit the authors flag before anyone else can. "The labels are derived from reports, which limits conclusions about visual correctness," the paper says. The audit shows the model is doing less with the image than it appears, not that the reports themselves are wrong. The direction, the authors add, "replicates in two further model lineages."

Substitutions also move continuous answer scores where the binary prediction does not change, suggesting the image signal is present but rarely decisive.