Capraro study: AI advice cuts user accuracy from 27% to 9%

A new paper covered by The Next Web is the sort of study that ought to give anyone deploying AI copilots a moment of pause. When participants had access to AI advice on visual detail questions drawn from films, their accuracy fell from 27% to 9%. Their confidence went the other direction, from 30% to 76%. Their willingness to say "I don't know" collapsed from 44% to 3%.

The three authors, Valerio Capraro at the University of Milano-Bicocca, Chiara Marcoccia at École Normale Supérieure, and Walter Quattrociocchi at Sapienza University of Rome, designed the test to isolate one thing. They picked Step 3.5 Flash, a model they knew was usually wrong on these questions, so any drop in human judgment could not be excused as sensible delegation to a reliable tool. Some participants who would have answered correctly on their own asked the AI and became wrong. Even offering monetary incentives to be accurate only nudged the numbers to 16% accuracy and 8% willingness to admit ignorance, both still well below the no-AI baselines.

"People became much worse, the accuracy was only one third, but they were twice as confident," Capraro told the outlet. His stronger claim is a metacognitive one, that "the mere availability of AI suppresses the cognitive habit of recognising what you do not know." Wharton researchers reached a related conclusion earlier this year, coining the term "cognitive surrender" to describe users accepting incorrect AI answers 80% of the time while reporting higher confidence than those working without AI.

The honest caveat is that this is one paper, using a deliberately weak model on a niche task, and the reporting does not give you the sample size, the participant demographics, or how the effect changes with more capable systems. Take the specifics as reported, not settled. What is worth watching is that the shape of the finding matches what Wharton saw independently. For anyone shipping AI assistants into schools, support desks, or knowledge work, the interesting design question is no longer how to make the model more helpful, it is how you preserve the user's ability to say "I don't know" when the model is confidently wrong.