Change.org AI tool reshaped petitions but didn't lift outcomes

An AI writing tool Change.org rolled out in late 2023 reshaped how petitions sound without changing how they do, according to a Nature Human Behaviour study that analyzed roughly 1.5 million petitions.

Cornell doctoral student Isabel Corpus, with co-authors Mor Naaman and Allison Koenecke at Cornell and Eric Gilbert at the University of Michigan, used a staggered launch as a natural experiment. The "write with AI" feature went live for users in the US, Great Britain and Canada on October 2, 2023, and did not reach Australia until December 15, eleven weeks later. That country gap let the team run a difference-in-differences comparison. The paper's verdict: AI access "substantially altered the lexical features of petitions and increased petition homogeneity but did not improve petition outcomes."

After the integration, petitions ran longer and leaned on formal imperatives such as "mandate", "implement" and "urge". They also drew slightly less engagement. The share of petitions reaching even one comment within 30 days of posting fell by about five percent against pre-rollout baselines.

"That was a surprise for us," Corpus told Phys.org, citing prior work in which AI-generated text had come across as "persuasive, it's believable and it can be creative."

The puzzle, in Naaman's reading of the data, is that the AI-assisted petitions had "the markers of what was successful in the past" without converting on them. Corpus suggests the bottleneck sits upstream of the words: people "sign a petition or engage with a cause because someone they trust, or an organization they trust, shares it with them." A separate analysis of 4,611 repeat petition writers found the same pattern within individuals, with their post-AI drafts underperforming their earlier work.