arxiv.org web signal

Study: GPT, Claude mine paper appendices for 50%+ of critiques

TL;DR

  • Latest GPT and Claude models draw 50% or more of their review weaknesses from the appendix, versus 8.4% for human reviewers.
  • Researchers examined 2,000 papers from ACL, EMNLP, COLM and ICLR and saw the same pattern across API, coding-agent and web interfaces.
  • The authors report the appendix bias is mainly a property of newer models and track human appendix-weaknesses rising to 27.60% in ICLR 2027 reviews.

The latest GPT and Claude models pull about 50% or more of their review weaknesses from the appendix of a paper. Human reviewers pull 8.4%.

That gap is the central finding of a preprint on arxiv by Saptarshi Ghosh, Mohammad Saim and Tianyu Jiang, who examined 2,000 papers submitted to ACL, EMNLP, COLM and ICLR. The abstract states it plainly: "the latest GPT and Claude models draw about 50% or more of their review weaknesses from the appendix."

The kinds of complaints differ too. LLM critiques of the appendix cluster on technical details: cross-checking numbers, asking for additional experiments and statistical validation on results already in the paper. Humans, when they do flag the appendix, object when essential sections have been pushed there.

The authors report the bias "is mainly a property of newer models". GPT-5 sits under 30% appendix focus; newer GPT and Claude releases run in the 60-70% range. The paper tracks human appendix-related weaknesses rising from around 10% across 2020-2024 to 27.60% in ICLR 2027 reviews, a shift the authors line up with recent model releases.

Three researchers we follow shared the preprint the same day.

Shared on Bluesky by 3 AI experts