Every tested LLM less diverse than web search, study finds

Every large language model in a new study is less epistemically diverse — that is, produces a narrower range of distinct real-world claims — than a basic web search, and the largest models come off worst. The paper, released with a companion Python package on GitHub, tests 27 LLMs on 155 topics covering 12 countries, generating 1.7 million responses and 70 million individual claims.

The abstract does concede progress. "Epistemic diversity has increased substantially over the past three years, a positive counter to recent diversity pessimism," it states, before the sharper finding: "every system is less diverse than a search baseline."

Two results cut against intuition. Retrieval-augmented generation lifts diversity, while "large models are counterintuitively less diverse than smaller ones." A third cuts across cultural context: "LLM parametric knowledge systematically reflects English over local-language knowledge for country specific topics."

Two of the researchers we follow posted the repo within a day of its release.

Which of the 27 systems ranked where, the per-country diversity gaps, and how much RAG helps in each context all sit in the full paper rather than the abstract; the repo publishes the claim-extraction package used to run the analysis, and lists Dustin Wright, Sarah Masud, Jared Moore, Srishti Yadav, Maria Antoniak, Peter Ebert Christensen, Chan Young Park, and Isabelle Augenstein as authors.