EvalResearchBench: Agent Evaluators Hit 75% Human Agreement
TL;DR
- AI evaluator agents matched human-researcher rankings on about 75% of candidate pairs, below the 91% ceiling set by humans disagreeing among themselves.
- The lowest-cost evaluator achieved the highest pairwise agreement with the nine human targets, beating more expensive models.
- The agents showed concrete failure modes including answer truncation and budget exhaustion, with human-designed benchmark samples remaining competitive.
The best AI evaluators ordered about 75% of candidate pairs the same way the human targets did. The ceiling, set by disagreements among those humans, was 91%.
That gap is the finding in EvalResearchBench, a new paper by Yaolun Zhang and colleagues asking whether AI agents can autonomously design the evaluation benchmarks that recursive self-improvement pipelines rely on. The authors benchmarked thirteen candidate models against nine human researchers.
'The best evaluators order about 75% of candidate pairs as the targets do,' the paper reports, against a '91% ceiling set by disagreements among the targets.'
Two side notes carry weight. Human-designed benchmark samples remained competitive against the agents. And the lowest-cost evaluator achieved the highest pairwise agreement: the cheap model, not the big one, lined up best with the researchers.
The abstract flags concrete failure modes in the agents: answer truncation and budget exhaustion. It does not list which thirteen models were tested or which domain the nine researchers worked in.
Originally reported by paper
Read the original article →Original headline: EvalResearchBench: AI Agents Reach 75% of Human-Benchmark Agreement on Self-Built Evaluation Suites — Short of 91% Ceiling