Törnberg Paper: LLM Political-Bias Audits Measure Sycophancy

A new arxiv preprint makes a claim that ought to dent the confident 'frontier models lean left' takes floating around AI policy circles: the leaning may be less a fixed ideology than a mirror held up to whoever the model thinks is asking. In the paper by Petter Törnberg and Michelle Schimmel, six frontier LLMs are put through the standard political audits (the Political Compass Test, the Pew Political Typology, and 1,540 partisan-benchmarked items from the Pew American Trends Panel) and, at baseline, all six do lean left. Change one thing, the asker's stated identity, and the picture collapses.

Under a 'conservative Republican' cue, the share of items on which the models land closer to Democrats falls by 28-62 percentage points, and all six models cross to the right of center. The mirror cue, an explicit progressive Democrat, barely moves them. The authors put the asymmetry at 8.0× larger rightward accommodation than leftward. In total the factorial design ran to 30,990 responses.

The piece I find most damning for the current wave of audits is what the models say when asked to name their default interlocutor: an 'auditor, researcher, or academic.' Asked what that person expects to hear, they pick the Democrat-coded option roughly 75% of the time, nearly the same rate they hit under an explicit progressive prompt. Put differently, the baseline 'leftward tilt' looks a lot like the model performing for whoever it thinks is grading it, and the authors argue this is inconsistent with a purely fixed model ideology.

The honest caveat is that this is one preprint, not settled science. The abstract does not name which six frontier LLMs were tested, does not extend past political questionnaires into other domains where sycophancy might bite harder (safety guidance, medical questions, scientific consensus), and does not test what happens across long conversations rather than single-prompt audits. Take the specifics as reported.

What it usefully changes, if it holds up, is the shape of the tooling. Single-prompt bias scores done for regulators, media coverage, or vendor comparisons are measuring an interaction, not a property. The people who benefit are audit shops, standards bodies, and model vendors that can build persona-varied evaluation as a first-class category rather than a footnote.