Study finds AI-text detectors 'neither accurate nor reliable'

The published verdict from a group of eight academic-integrity researchers is blunter than the usual 'more work needed' hedge. In a study published in the International Journal for Educational Integrity, Debora Weber-Wulff and co-authors put 14 AI-text detectors, 12 public tools plus the commercial systems Turnitin and PlagiarismCheck, through a controlled test and conclude that 'the available detection tools are neither accurate nor reliable.'

The numbers behind that verdict are the part worth staring at. According to the paper, every tool scored below 80% accuracy and only five cleared 70%. Six of the fourteen produced false positives, which is the failure mode that matters most for universities, because it means a human-written essay flagged as AI. Thirteen of the fourteen produced false negatives on at least some AI-generated documents. The overall bias, the authors write, is 'towards classifying the output as human-written rather than detecting AI-generated text,' and accuracy degraded further once the AI output was manually edited, machine-paraphrased, or machine-translated.

If you run a university, that combination is a policy problem, not a tool problem. A detector that both misses obvious cheating and randomly convicts real students cannot underwrite a disciplinary process. The paper does single out one commercial exception in the false-negative test: only Turnitin correctly classified all documents in the AI-generated classes tested. That will show up in procurement conversations, though passing one class of tests is not a general endorsement.

The honest caveats are worth naming. The retrieved reporting does not give sample sizes per tool, which ChatGPT version produced the test documents, or whether the false-positive risk for non-native English writers was isolated as its own finding, though the paraphrasing and translation results imply direction. Take the per-tool numbers as reported, not settled.

The forward-looking read is that this pushes the burden back onto assessment design. If detection is the wrong lever, the workable ones, process-based assessment, oral defenses, in-class writing, portfolios of drafts, are the ones that do not need a detector to work.