arxiv.org web signal

ChatGPT power users outscore AI-text detectors, ACL paper finds

By Published Updated

TL;DR

  • Five expert annotators, voting as a group, misclassified only 1 of 300 non-fiction articles across GPT-4o, Claude and o1 outputs.
  • The human group outscored most commercial and open-source detectors even when AI text was paraphrased or 'humanized,' the authors report.
  • Experts leaned on 'AI vocabulary' and softer cues like formality, originality and clarity that automatic detectors struggle to assess.

Five frequent ChatGPT users, voting as a group, misclassified just one of 300 non-fiction English articles when asked which were written by humans and which by GPT-4o, Claude, or o1. That performance outscored most of the commercial and open-source detectors the authors evaluated, even against evasion tactics like paraphrasing and 'humanization,' according to an ACL 2025 paper from Jenna Russell, Marzena Karpinska and Mohit Iyyer.

The annotators got no specialized training. The authors describe them simply as people who 'frequently use LLMs for writing tasks,' and report that this group 'excel at detecting AI-generated text, even without any specialized training or feedback.'

What the humans picked up on is the more interesting part. The paper reports they 'rely heavily on specific lexical clues ("AI vocabulary"),' but also notice 'more complex phenomena within the text (e.g., formality, originality, clarity) that are challenging to assess for automatic detectors.' Two of the researchers we follow at AI Weekly had already circulated the paper.

The abstract names GPT-4o, Claude and o1 as the generators tested and refers to commercial and open-source detectors in aggregate, without listing which detectors were evaluated or publishing per-detector numbers. The authors say they 'release our annotated dataset and code to spur future research into both human and automated detection of AI-generated text.'

Shared on Bluesky by 2 AI experts

Originally reported by arxiv.org

Read the original article →

Original headline: People who frequently use ChatGPT for writing tasks are accurate and robust detectors of AI-generated text