Baumann et al: AI peer reviewers hivemind and get gamed

Peer review is drowning in submissions and there is an obvious temptation to hand it to a large language model. A new position paper accepted as an oral at ICML 2026's Position Paper Track argues that today's AI systems should not be doing this, and puts numbers behind the objection. Joachim Baumann, Jiaxin Pei, Sanmi Koyejo and Dirk Hovy ground the argument on arxiv in real ICLR 2026 submission data, and the pattern they describe is unflattering to the 'just let a model do it' pitch.

The first finding is what they call a hivemind effect. AI-generated reviews look more like each other than human reviews do, both within a single paper and across different papers entirely, with within-paper similarity running 8.7% to 9.8% higher and cross-paper similarity 4.1% to 39.8% higher. On the ICLR 2026 corpus the authors examined, fully AI-generated reviews had a mean within-group similarity of 0.486 against 0.467 for reviews with any human contribution. In practical terms, the range of perspectives a paper gets from AI reviewers is narrower than what it would get from people, which erodes the whole point of soliciting multiple reviewers in the first place.

The second finding is that scores can be trivially inflated by rewriting the paper. In the authors' experiments, zero-shot LLM rewrites boosted AI review scores by +0.45 on average (p<0.0001), and laundered papers converged toward each other in style. Sitting alongside this: on 8,015 papers that received both human and AI reviews, averaged human scores predicted acceptance at AUC 0.822 while averaged AI scores managed only 0.710. Human reviewers are noisy, but the machines here are both blander and easier to fool.

The honest caveat is what the paper does not claim. It is not saying AI has no place in the review pipeline; it is saying general-purpose LLMs deployed without domain-specific evaluation are not the answer, and it calls for 'a science of peer review automation' rather than a ban. The scope is also one conference cycle, and what the reporting does not give you is a design for that science: which metrics beyond gameability and diversity would need to clear a bar, or what a certified reviewer bot would even look like in production.

For conference chairs and journal editors, the useful takeaway is that adopting AI-assisted review needs its own benchmark suite before it goes near real decisions. For the LLM shops eyeing this as a market, the paper reads as a fairly loud invitation to build the eval harness, not the reviewer.