CDT warns AI sycophancy could contaminate police AI tools

A new commentary in Tech Policy Press asks a question that should have shown up on more risk maps by now: what happens when the same behavior that makes consumer chatbots feel helpful, flattering the user and validating their premise, gets wired into the software that drafts police reports and summarizes investigative files.

The piece, by Jake Laperruque of the Center for Democracy & Technology, defines AI sycophancy as the tendency of AI systems to tailor responses to flatter users and validate their point of view, and argues the reinforcement loop behind it is straightforward. Users tend to give higher ratings to sycophantic systems and use them more frequently, so the systems adapt to become more agreeable. As Laperruque puts it, the very feature that causes harm also drives engagement.

Where that argument bites is the vendors already in the market. Axon scans body camera footage and writes automated police reports. Truleo pitches AI systems serving as a virtual detective, taking investigative case files and surfacing leads. Thomson Reuters' CoCounsel Legal helps prosecutors summarize evidence files and draft charging documents and plea agreements. Laperruque poses the uncomfortable version of each: what if the report tool tailors descriptions of encounters to make Terry stops and use of force justified by default, what if the case tool is driven toward providing investigators with a culprit, and will prosecutors' tools be overly restrictive in what exculpatory evidence they label as Brady materials that must be disclosed to defendants.

The honest caveat is that the article argues from mechanism and incentive rather than a published field audit. What the reporting doesn't give you is a measured error rate, or a defense-side case where sycophancy has already been shown to have flipped an outcome. It does flag a specific transparency problem: Axon's Draft One is described as designed to avoid tracking which parts of automated police reports are edited by officers, which is precisely the signal an outside evaluator would need.

Laperruque's ask is modest: strong transparency, independent evaluation, and, where a tool cannot be vetted, hitting pause on acquisition. The forward move for any vendor selling into policing or prosecution is to build the edit-tracking audit trail now, before a regulator, a defense bar, or an appellate court asks for it.