Change.org AI tool changed petitions, didn't improve outcomes

Change.org added a 'write with AI' tool, and petitions on the platform got more homogeneous and lexically different. They did not get more successful.

That's the finding of a new paper in Nature Human Behaviour by Isabel Corpus, Allison Koenecke and Mor Naaman at Cornell Tech, with Eric Gilbert at the University of Michigan. They collected 1.5 million petitions and used a difference-in-differences analysis to isolate what the AI tool did.

'AI writing tools can profoundly reshape online content, their practical utility for improving desired outcomes may be less beneficial than anticipated and introduce unintended consequences such as content homogenization,' the authors write.

To rule out a composition-shift artifact, the team ran a second analysis on 4,611 users who had posted petitions both before and after the tool arrived. The pattern held: writing changed, outcomes did not.

The paper's abstract reports the direction of that outcome effect, not its magnitude. How large the gap is between AI-assisted and non-AI petitions is not stated there.