Oxford LLM study: 90% of political social-science papers lean left

A doctoral candidate at Oxford pointed a large language model at roughly 600,000 social-science abstracts from 1960 to 2024 and asked it to score each one on a left-right political scale. The methodology is worth paying attention to on its own, separate from the headline finding, because it is one of the more visible attempts to use an LLM as a measurement instrument for content analysis at a scale that would not be feasible with human coders.

The study by James Manzi, published in Theory and Society, ran 599,194 English-language abstracts across 367 journals in 11 disciplines through what he describes as a fixed rubric anchored to the U.S. political spectrum as it existed in 2025, with a 0-to-10 scale where 0 is far right and 10 is far left. Around 180,311 of those abstracts were judged politically relevant, and roughly 90 percent of that subset leaned left. According to a write-up in PsyPost, every one of the 11 disciplines examined, from anthropology and economics to gender studies and sociology, sat left of center in every year of the period, and all of them moved further left between 1990 and 2024. Fields with stronger leftward orientations tended to show less political diversity, and Manzi attributes the drift primarily to new contributors entering the publication stream rather than to individual scholars shifting their views over time.

The obvious challenge to a study like this is that LLMs themselves have well-documented left-of-center leanings, so asking one to score political content risks measuring the instrument as much as the subject. Manzi's response, per a summary in Free the Inquiry, was to validate the model's outputs against blinded think-tank texts and to check stability across repeated runs. That is the kind of calibration step this class of work will need to make routine if LLM-as-coder is going to become a real tool for the social sciences rather than a novelty.

The honest caveat is one Manzi flags himself. He told reporters that "there's nothing here that describes the causal mechanism," and that the findings do not constitute "dispositive evidence of bias." What the reporting does not give you is the exact rubric and prompt text, a full discipline-by-discipline time series, or an independent replication with a different frontier LLM. Those are the pieces outsiders would need to judge how much of the signal is really in the abstracts and how much is in the model. The interesting AI-side takeaway is that a single researcher can now credibly attempt a corpus study at this scale, and the question of whether to trust the tool is starting to hinge less on intuition and more on the sort of validation checks Manzi ran.