Change.org's AI tool made petitions longer, not stronger

Petitions written after Change.org's 'write with AI' tool launched were 5.33 percentage points less likely to clear 10 signatures than petitions written before it existed, according to a study published in Nature Human Behaviour by researchers at Cornell and the University of Michigan.

The paper, led by Cornell's Isabel Corpus with Eric Gilbert, Allison Koenecke and Mor Naaman, analyzed 1.5 million petitions. Change.org rolled out the tool first in the United States, Great Britain and Canada starting October 2, 2023, and gave Australian users access eleven weeks later. That stagger let a difference-in-differences design compare the platform against a near-identical control.

The AI-assisted petitions were, on every lexical measure, more polished. Median word count rose by 53.63 words, a 49% jump from a 110-word baseline. Flesch-Kincaid grade level climbed 1.49 grades. Pairwise similarity between petitions rose 0.056, a 23% increase in homogeneity. The authors conclude that 'in-platform AI access significantly altered the lexical features of petitions and increased petition homogeneity, but did not improve petition outcomes.'

Corpus told Phys.org the result caught the team off guard: 'That was a surprise for us.' Naaman noted the AI-polished petitions 'had the markers of what was successful in the past.' One possible read he offered: 'People didn't have their AI radars as high in 2023.'

The signature effect was significant at p=.001. A second metric, whether a petition drew at least one comment in 30 days, fell by 2.51 percentage points but did not clear significance (p=0.187). Two researchers in our Who's Who tracker shared the paper the day it dropped.