Torralba and Weiss derive contrastive learning's optimal filters

A new ICML 2026 paper by Antonio Torralba and Yair Weiss, posted to arXiv in July, takes on a question that has hung around self-supervised vision for years: why does contrastive learning with simple augmentations produce useful representations at all. The answer they offer is not another benchmark story. It is an analytical result, an actual computation of what the optimal representation looks like under a contrastive loss, given any image dataset with stationary statistics.

The specific claim is precise. For certain augmentations the optimum can be attained by a CNN whose first-layer filters are sinusoids, followed by a pointwise nonlinearity, global average pooling, and a final linear layer that performs partial whitening. Even for more complicated augmentations, the optimal weights in such CNNs are still sinusoids. The frequencies and their weights can be computed with a simple waterfilling algorithm from the dataset's expected power spectrum, which is the kind of construction people who came up through signal processing will recognise immediately.

Why this matters if you do not build vision encoders yourself: the field has largely known contrastive pretraining works and has justified it with empirical scaling curves and post-hoc intuitions. A derivation that pins down what the optimum actually is gives researchers a real object to argue about, to check trained models against, and to teach from. The empirical half of the paper reports that CNNs trained with SGD on different image datasets do spontaneously learn sinusoids in their first layer and do perform partial whitening, which lines up with the theory.

The honest caveats are worth naming. The result is derived under stationary-statistics assumptions and formulated for CNNs, not for the transformer-based image encoders many modern contrastive systems use, and the abstract does not name which specific datasets were tested or quantify the gap between predicted and observed filters. As a theoretical foothold for a technique that has been mostly empirical, though, it is the kind of result the community has been waiting on, and the natural next step is extending the framing beyond CNNs and beyond image-only augmentations.