ICML study finds nearly half of LLM benchmarks saturate

Something modest but useful landed at ICML 2026: a systematic study of AI benchmark saturation that actually measures how quickly benchmarks stop being useful. The authors analyzed 60 language model benchmarks against 14 properties tied to saturation, and reported that nearly half of them exhibit it, with rates rising as the benchmarks age. Two of the experts in our Who's Who directory shared this paper, which is part of why it caught our eye.

The more interesting finding is what protects against saturation. The paper says resilience is impacted by expert-curation, not by public test data. That inverts a common intuition, that a hidden test set is the main defense against overfitting or contamination. The paper's claim is that how carefully the questions were constructed matters more than whether the answers leaked.

For anyone using public leaderboards to pick a model or ship a capability claim, this is a small but real caveat. If half of the benchmarks in circulation are already saturated, then two models tied at the top of one of them may not actually be tied at anything meaningful. The paper argues that design choices can extend benchmark longevity, which is another way of saying that a lot of benchmark work is disposable by construction.

The abstract does not name which specific benchmarks in its set of 60 are the saturated ones, and it does not spell out how the 14 properties were weighted against each other. Those details matter if you want to translate the finding into a purchasing or evaluation policy, and they are the reason to read the full paper before quoting the headline number.

If the recommendation lands, the winners are benchmark designers willing to spend on expert curation, and the buyers who learn to prefer their outputs.