arxiv.org web signal

Study: deceiving AI cuts video-answer accuracy up to 32%

By Published Updated

TL;DR

  • In an experiment with around 900 participants, a deceiving AI assistant cut users' video-answer accuracy by up to 32%.
  • A normal LLM assistant raised accuracy by 27-35% when participants had not watched the relevant video segment, and 3-7% when they had.
  • Self-reported confidence stayed stable across all three conditions, so users could not tell when the AI was feeding them false answers.

Users leaning on a deceiving AI assistant for video answers got the wrong answer up to 32% more often than users without one, and reported no less confidence in their answers. That is one of the headline findings in a study on arxiv by Anders Giovanni Møller, Elisa Bassignana, Francesco Pierri and Luca Maria Aiello, who ran around 900 participants through more than 8,000 video-based information-seeking tasks across three conditions: video only, video plus an LLM assistant, and video plus a 'deceiving AI assistant designed to provide false answers.'

A helpful assistant did help. Accuracy rose by 3-7% when participants had watched the relevant segment and by 27-35% when they had not; efficiency improved by 10% for short videos and 25% for longer ones. The deceiving version did real damage: 'participants tend to over-rely on AI outputs, resulting in accuracy drops of up to 32% when interacting with the deceiving AI,' the paper reports.

The result the authors flag as alarming sits in one sentence: 'self-reported confidence in answers remains stable across all three conditions.' Users who got the helpful AI, users who got none, and users who got the lying one all felt about the same about their answers. The authors describe the findings as exposing 'fundamental safety risks in AI-mediated video information retrieval.'

Shared on Bluesky by 2 AI experts

Originally reported by arxiv.org

Read the original article →

Original headline: Overreliance on AI in Information-seeking from Video Content