Bekkerman: 2.5M Papers Show LLMs Haven't Transformed Science
TL;DR
- A 2.5-million-paper analysis across five disciplines from 2015 to 2025 found most pre-2023 scientific-practice trends continued with limited detectable breaks after LLMs.
- Computer-science coauthor teams in Asia expanded at an accelerated rate, one of the few significant breaks the authors flag.
- Apparent declines in biomedical topic diversity traced back to standardization in article formats rather than substantive research change.
A group of four researchers combed 2.5 million full-text scientific papers from 2015 to 2025 and found that, aside from frequency shifts in some words, the late-2022 arrival of large language models barely registers in how science is being written.
Ron Bekkerman, Nichole Li, Bahareh Kargar and Mark Somers posted the paper on arXiv on November 17. They traced "lexical, structural, epistemological, and social dimensions" across five disciplines over the decade and report that "most pre-2023 trends continued with limited detectable breaks."
Two exceptions stood out. Computer-science coauthor teams in Asia kept expanding, and that growth "accelerated significantly." Biomedical topic diversity looked like it was contracting, until the authors traced the drop to standardization in article formats rather than substantive change in what researchers were studying.
The authors frame the work as measurement infrastructure, not verdict: "Our study establishes an empirical foundation for monitoring the health of global science as LLM adoption deepens." The abstract names neither the five disciplines nor the specific words that moved.
Originally reported by arxiv.org
Read the original article →Original headline: Arxiv Study of 2.5M Papers 2015-2025 Finds LLMs Haven't Yet Transformed Science