BenchBenchBench audits AI benchmark-authoring evaluators

A small repo worth pulling if you follow how AI evaluations get graded. BenchBenchBench asks a narrow but important question: when a public score rates an AI model's ability to generate benchmarks, does that score actually predict how well its benchmarks perform on evidence the scoring system did not use?

The setup is a decision-level holdout audit. The repo bundles deterministic tasks with frozen model submissions from July 2026, scoring code, and a reproducible pipeline that runs without API credentials. Against composite targets, the full evaluator reports a Spearman correlation of 0.945, pairwise accuracy of 90.8%, and utility recovery of 98.9%. Restrict the scoring to hidden-only evidence and the numbers slide meaningfully, to 0.580 correlation, 73.3% accuracy, and 93.1% utility recovery.

Why that gap matters: if the score you use to pick the 'best benchmark-authoring model' does not generalize past the evidence the score itself used, any leaderboard ranking of benchmark-generator models is a shakier procurement signal than it looks. As older benchmarks saturate, using models to generate the next generation of evals is only useful if the scoring holds up out of sample, and a drop from 90.8% to 73.3% on hidden-only is the kind of gap that would drive different decisions.

The honest caveat, and the repo says this itself, is that the run was 'retrospective and was not preregistered,' and researchers inspected holdout behavior during development. The authors position the artifact as 'a reproducible conformance audit, not as confirmatory evidence from a researcher-blind holdout.' The project also credits Rohit Krishnan's May 2026 essay 'Introducing BenchBench,' which 'publicly framed benchmark creation as a model capability,' and notes BenchBench-M is a separate implementation.

What the README does not tell you is which model submissions were used as authors, how the holdout split was constructed to avoid inspection leakage, or what an independent preregistered replication would look like. Those are the follow-ups. For anyone building or buying evaluation platforms, the design pattern (decision-level holdouts as a defense against evaluator overfit) is the takeaway even before the specific numbers are re-run.