New Metric Finds Benchmark Contamination Fixes Overstated

Benchmark decontamination is the thing everyone quietly relies on to keep leaderboard numbers meaningful, and a new arXiv preprint from Ruijie Hou, Yueyang Jiao, Zhao Wang and Yingming Li argues that the standard way of checking whether decontamination worked has been giving the community credit it did not earn.

The current metric, Gap of Aggregate Performance or G-AP, compares a model's average score on a contaminated benchmark to its average on a clean twin. The paper's argument, quoted directly, is that this approach 'cannot characterize individual question performance through binary correct/incorrect labels' and that 'averaging before differencing masks over- and under-suppression.' In other words, if a mitigation over-suppresses on some questions and under-suppresses on others, those errors happily cancel inside a single averaged gap, and the mitigation looks cleaner than it actually is.

Their replacement, SA-PPG, works one question at a time. It samples completions to estimate a solve probability, differences that against a clean model per question, and then aggregates within groups defined by the clean model's own solve probability. Under that lens the authors report that 'prior strategies' restoration is substantially overestimated.' They also propose their own mitigation, RailCap, which intervenes during generation. The rule they describe is that 'whenever a sample falls back onto the greedy trajectory, the next trajectory token is capped to the runner-up,' and, unsurprisingly for a paper's own method, RailCap comes out on top on SA-PPG.

Some things worth holding lightly. The abstract never puts a number on how much prior mitigation is overestimated, never names the contaminated models or benchmarks used in the experiments, and this is an unrefereed arXiv preprint. RailCap's edge is also measured under a metric its own authors designed, which is a familiar shape for methods papers and deserves independent replication before anyone treats the leaderboard as re-ranked.

If the core finding holds up, the interesting downstream move is for benchmark maintainers and third-party evaluators to re-audit the contamination claims already baked into model cards, and for model builders who did the harder decontamination work honestly to see that effort start to show up in the scores.