NeurIPS paper audits 25 AI benchmarks for difficulty bias

A NeurIPS 2026 paper by Yifan Sun, Naicheng Yu, Jiawen Gong, Jingyan Shen and Huan Zhang reframes AI benchmarks as empirical probability measures over difficulty, then audits 25 popular benchmarks across 7 domains and reports that "10 out of 25 benchmarks have DDM≥0.10" and "8 out of 25 benchmarks have Max Gap≥0.10" — their thresholds for substantial difficulty-weighting bias and for leaving regions of the difficulty axis uncovered.

The framing is in the opening line of the abstract: "A benchmark is not merely a collection of test questions; it naturally induces an empirical measure over difficulty." From that view the authors formalize two failure modes, *measure mismatch* and *coverage gaps*, and argue that "many benchmarks implicitly overweight certain difficulty regions, so their leaderboard conclusions should be interpreted with caution."

The paper also proposes a fix rather than only a diagnosis. The authors describe BenchFill as "a simple difficulty-targeted item rewriting algorithm that effectively reduces local coverage gaps," with an official implementation posted to the ASTRAL-Group GitHub repository linked from the arXiv page.

The abstract does not name which benchmarks fall into the 10-of-25 or 8-of-25 groups, and the headline counts depend on the authors' item-difficulty estimator, which the summary does not fully detail.