arxiv.org web signal

CLEVR-HOPE: diversity, not volume, closes VQA systematicity gap

TL;DR

  • A new diagnostic dataset called CLEVR-HOPE is introduced to measure the gap between reasoning on seen and unseen combinations of object attributes.
  • Scaling the quantity of training data did not reduce the systematicity gap in visual question answering, the authors report.
  • Increasing the number of distinct attribute-type combinations in training did reduce the gap, suggesting diversity drives compositional generalization.

Training data diversity does more for compositional reasoning than training data quantity, according to a paper by Ian Berlot-Attwell, Kumar Krishna Agrawal, A. Michael Carrell, Yash Sharma and Naomi Saphra.

The authors frame the problem as the "systematicity gap in visual question answering: the performance difference between reasoning on previously seen and unseen combinations of object attributes." To measure it, they introduce a diagnostic dataset called CLEVR-HOPE.

The headline result is blunt. "We find that the systematicity gap is not reduced by increasing the quantity of training data, but is reduced by increasing the diversity of training data," the authors write. They add that "the more distinct attribute type combinations are seen during training, the more systematic we can expect the resulting model to be."

The abstract publishes no per-model numbers and does not name the architectures evaluated, so the size of the effect is not stated in the material released here.

Shared on Bluesky by 1 AI expert