LeCun paper: in 100+ dimensions, models always extrapolate

Interpolation almost surely never happens in more than 100 dimensions. That is the finding of a preprint by Randall Balestriero, Jerome Pesenti and Yann LeCun, posted to arXiv in October 2021. Using the geometric definition, where a sample counts as interpolated only when it falls inside or on the boundary of the training set's convex hull, the authors argue any real high-dimensional test point almost always falls outside that hull, so models routinely described as 'interpolating' are in fact always extrapolating.

They frame the work against two widely held beliefs. The first, in their words, is that 'state-of-the-art algorithms work so well because of their ability to correctly interpolate training data.' The second is that 'interpolation happens throughout tasks and datasets.' Both, they write, are misconceptions.

The stated consequence is that the results 'challenge the validity of our current interpolation/extrapolation definition as an indicator of generalization performances.' A large body of theory that leans on interpolation to explain why deep networks generalize would then need a different account.