Equivariance still pays off at scale, arxiv paper finds

A quietly useful arxiv paper from Brehmer, Behrends, de Haan and Cohen asks a question the field has been arguing about for a few years now: if you make your models big enough, do you still need to bake problem-specific symmetries into the architecture, or does scale wash that advantage out. Their paper says the advantage does not wash out.

They ran a controlled comparison on a rigid-body interaction benchmark, sweeping model size, training steps and dataset size for both an equivariant transformer and a plain non-equivariant one. The headline finding is that the equivariant version wins at every compute budget they tested, and that the scaling curves follow power-law patterns for both, but with the equivariant one sitting above. That is the part that matters for anyone who has been assuming the bitter lesson quietly settled this debate in favour of 'just add more data and parameters'.

The more interesting nuance is on data augmentation. Non-equivariant models trained with augmentation can close the data-efficiency gap, but only given sufficient epochs. So the two approaches are not straightforwardly equivalent even in the limit, they just meet on a different axis, and the compute-optimal split between model size and training duration ends up looking meaningfully different between the two families.

The honest caveat is that this is one benchmark and one architecture family. Rigid-body dynamics is exactly the kind of problem where geometric structure is obvious and cleanly specified, and the paper does not tell you how far the result travels to molecules, images, or other symmetry groups, nor at what epoch count augmentation actually catches up in practice. Take it as an empirical data point against 'equivariance is obsolete at scale', not as a settled compute-optimal recipe.

For teams still building on limited hardware, in physics simulation, robotics, protein and molecular modelling, the practical read is that architectural inductive bias is not free lunch that scale makes irrelevant. It is still a lever, and if this paper generalises, it is a lever that gets you a better result per FLOP.