Change.org AI tool reshaped petitions but not success rates

A new study of 1.5 million Change.org petitions finds that giving writers an in-platform "write with AI" tool reshaped how petitions sounded without making them more likely to succeed. The paper by Cornell's Isabel Corpus, Mor Naaman and Allison Koenecke with Michigan's Eric Gilbert was published in Nature Human Behaviour on September 30.

The team took advantage of an 11-week staggered rollout: Change.org switched on the AI assistant in the U.S., Great Britain and Canada first, and in Australia later, letting them run a difference-in-differences analysis across 1.5 million petitions. At the platform level, "in-platform AI access substantially altered the lexical features of petitions and increased petition homogeneity but did not improve petition outcomes," the paper reports. Petitions trended longer and leaned on formal imperatives such as mandate, implement, and urge.

Early engagement moved the wrong way. The share of petitions reaching their first comment within 30 days fell by about 5% after the tool went live.

That was unexpected, because the AI drafts carried the surface features of winning petitions. "We saw in the data that those [AI-enhanced] petitions had the markers of what was successful in the past," says co-author Mor Naaman. Lead author Isabel Corpus notes that prior work found AI-generated text to be "persuasive, it's believable and it can be creative, sometimes more so than people writing alone." On Change.org, that pedigree did not translate into signatures or comments.