AI-on-AI review training triggers 'judgment collapse,' study finds

A paper posted on arxiv reports that when large language models trained on peer reviews are used to generate new reviews, and those synthetic reviews feed the next generation of reviewer models, the successor models produce noticeably narrower judgments. The authors, Sy-Tuyen Ho, Minghui Liu and Furong Huang, call the pattern "scientific-judgment collapse."

Their setup is deliberately clean. Starting from Llama 3.1 8B, they fine-tuned a base reviewer on official ICLR reviews from 2018 through 2023, then trained four successor models on ICLR 2024 data with, in their words, "systematically varied mixtures of official and model-generated reviews." The finding, quoting the abstract: "introducing synthetic reviews compresses rating distributions and reduces both same-paper and corpus-level semantic diversity."

The authors also release TrustReviewer, an open-source system that intervenes twice. At training time, the core reviewer is fitted "on a curated corpus designed to reduce low-quality and semantically degenerate supervision." At inference, "paired activation steering aims to further mitigate residual tendencies toward collapsed judgments without further training or additional expert annotation." The paper describes its scope as studying "one step of this feedback loop in a controlled setting," not measuring any conference in production.