Change.org AI tool reshaped petitions, didn't lift outcomes

Change.org added a 'write with AI' button in late 2023. The petitions got longer and more similar to each other. Fewer of them picked up even a single comment.

That is the finding of a new paper in Nature Human Behaviour by Isabel Corpus, Allison Koenecke and Mor Naaman of Cornell and Eric Gilbert of the University of Michigan. They collected 1.5 million petitions and ran a difference-in-differences analysis across the staggered rollout. The tool arrived in the U.S., Great Britain and Canada first, and in Australia 11 weeks later.

The vocabulary moved in a specific direction. 'Petitions started calling for people to "implement," "mandate" or "urge," whereas before we rarely saw that these words were used,' Corpus told Digital Information World. The AI appeared to be echoing the phrasing of historically successful campaigns.

That echo did not carry the success with it. 'We saw in the data that those [AI-enhanced] petitions had the markers of what was successful in the past,' Naaman said. The share of petitions that reached one comment after 30 days fell by around 5% relative to the pre-AI baseline. A separate analysis of repeat petition writers, who posted both before and after the rollout, confirmed the pattern.

From the authors' own abstract: 'AI writing tools can profoundly reshape online content, their practical utility for improving desired outcomes may be less beneficial than anticipated, and introduce unintended consequences like content homogenization.'