Change.org AI writing tool reshaped petitions, didn't lift outcomes

A natural experiment on Change.org found that giving petition writers an in-platform AI writing tool reshaped the words they used and made petitions more homogeneous, yet did not improve whether those petitions drew comments or signatures, and on some measures made them do worse. The analysis, by researchers at Cornell and the University of Michigan, covers 1.5 million petitions over the staggered 2023 rollout and is published in Nature Human Behaviour.

Change.org pushed the writing assistant live between October 2 and December 15, 2023, first in the United States, Great Britain and Canada, and 11 weeks later in Australia. That lag let the authors run a difference-in-differences design using the delayed country as a control. The share of petitions reaching at least one comment within 30 days dropped by roughly 5% after the tool went live, and the share reaching ten signatures did not improve against the pre-AI baseline.

"That was a surprise for us," lead author Isabel Corpus told Cornell's news office, noting that prior work had suggested AI-generated text is "persuasive, it's believable and it can be creative." Co-author Mor Naaman added that in the data, "those [AI-enhanced] petitions had the markers of what was successful" without converting into action. Petitions started leaning on verbs like "implement," "mandate" and "urge"; Corpus said the team "also saw a lot of homogeneity in the language of the titles." Two of the AI researchers we track circulated the paper the week it ran.

The paper's own framing is cautious: AI writing tools "can profoundly reshape online content," but "their practical utility for improving desired outcomes may be less beneficial than anticipated." Naaman flagged a timing caveat too, telling Cornell that in 2023 "people didn't have their AI radars as high" as they do now.