Princeton audit: ChatGPT memory tilts election answers by party

Turning ChatGPT's memory on makes its election answers more partisan, and the extra slant leans toward whichever party the user already belongs to.

That is the headline finding from a field audit by Princeton's Center for Information Technology Policy under its Algorithms+Elections Project. Rather than pound the OpenAI API, the researchers recruited 160 US ChatGPT users across five Senate battleground states (Texas, Ohio, Michigan, Iowa and Maine) and had them ask identical election questions from their own accounts between October 31 and November 3. Each question was posed four times: twice with memory enabled, twice with it disabled. Two AI judges from rival labs, one from Anthropic and one from OpenAI, rated every answer on a scale from -3 (favors the Democratic side) to +3 (favors the Republican side).

The average difference between personalized and non-personalized answers was 0.12 standard deviations, p < 0.001. Personalized answers moved 0.10 SD toward each user's party on average, p < 0.05. Democratic users' answers slid -0.11 SD, Republican users' slid +0.08. Local-news questions showed the largest gap at 0.24 SD, with the two versions of an answer differing by more than a point on the seven-point scale 38% of the time.

The slant is subtle. Only 2 of 576 voting-advice answers contained a direct endorsement.

The authors call these "preliminary analyses due to their timeliness" and say a fuller report with additional experiments is coming. They close with a line aimed squarely at the audit community: "we hope the results add nuance to this conversation by highlighting the limits of simple API-driven audits and the need to study chatbots as people actually use them." No individual byline appears on the post.