ArXiv's review-paper LLM ban misses six-times-larger tail

ArXiv banned unpublished CS review papers, citing a high prevalence of LLM-generated content, without publishing the numbers behind the decision. A preprint from Yanai Elazar and Maria Antoniak fills that gap, and the numbers complicate the policy more than they confirm it.

Using two detection methods, a statistical Alpha Estimator built on adjective occurrence patterns and a commercial transformer-based classifier called Pangram, they report adjusted cohort-level estimates of 21.4% for review papers versus 14.0% for non-review papers. So the ban's premise, that review categories carry a higher rate, holds up. What does not hold up is the scoping. In absolute counts, there are almost six times more non-review papers generated by LLMs than review papers, meaning the volume of the problem sits in the categories the policy leaves alone.

The distributional effect is uneven in a way the announcement did not flag. The authors estimate the Computers & Society subdiscipline could see cuts of 50% under the ban, more than any other CS domain they measured. That is a large hit to one community absorbed to catch what the same paper implies is a smaller slice of the overall LLM problem.

The honest caveat is that the paper does not report false-positive rates for either detector on human writing that happens to look polished, does not resolve where LLM-assisted ends and LLM-generated begins, and does not tell you how ArXiv will respond to its own evidence. Detectors of this kind, both statistical and commercial, are still working products, and this is a preprint that has not been peer-reviewed.

What is useful is the framework itself. The authors released their code, and the pairing of a cheap statistical estimator with a commercial classifier gives program chairs, journals, and ArXiv itself a way to check assumptions with numbers rather than intuition. If the review-paper ban is the start of a broader moderation posture rather than the end of it, this preprint is the closest thing to a receipt.