Researchers Deploy 1,120 AI Agents to Audit X's Feed at Scale

Independent auditing of recommendation systems has always sat between two bad options. Studies with real users capture how the feed actually behaves but are costly and hard to control. Sock puppet accounts scale, but their scripted behavior isn't lifelike enough to trust the results. A new arXiv paper from Alessandro Morosini, Sarah H. Cen, Andrew Ilyas and colleagues argues the missing piece is generative AI agents running as synthetic users.

The setup is straightforward. The team deployed 1,120 agents on X shortly after the 2024 U.S. election, spanning 14 personas and three counterfactual conditions, collecting over 200,000 content exposures. Each agent is instantiated with a fixed persona grounded in demographic and political survey data, then reasons about the platform's content and chooses actions. Because behavior stays constant within a persona while platform-visible signals like age, gender, or location can be experimentally perturbed, the design lets the authors compare how the same synthetic user is treated when only their attributes change. They call this counterfactual auditing.

The headline finding is that X's algorithmic feed amplifies toxic, polarizing, political, and right-leaning content relative to the chronological feed, with amplification varying sharply by user ideology. The demographic story is messier: pooled effects across personas are largely null, while subgroup effects vary substantially. That's a useful reminder that averaging across users can hide the actual signal.

The honest caveat is that a fixed-persona LLM is still a proxy for a real user, and any audit like this inherits whatever biases the underlying model brings when it reasons about what to click or repost. The paper also studies one platform, in one country, during a specific political window. What the reporting doesn't give you is a comparison against other feeds like TikTok, Instagram, or YouTube, or whether the same pattern holds outside election season.

Still, the direction is what matters. If black-box audits can run at 1,120-agent scale for a fraction of what a real-user panel costs, regulators, journalists, and outside researchers get a much cheaper way to keep pressure on the platforms they can't see inside.