Ullman's illusion-illusions fool GPT-4o, Claude 3, Gemini

The reliable way to check whether a system actually sees is to show it something plain and ask if it tells the truth. That is the move behind a short Harvard paper on arXiv by Tomer Ullman, and it lands harder than most benchmark write-ups because the test is so cheap.

Ullman flips the usual optical-illusion probe on its head. Rather than checking whether vision language models fall for the same illusions humans do, he shows them 'illusion-illusions': images that look like famous illusions but are not. Perfectly reasonable ducks where a classic illusion would put an ambiguous duck-rabbit. Crooked lines that truly are crooked. Circles that seem to have different sizes because they, in fact, are of different sizes.

The result is that the three leading models that correctly recognize real illusions, described in the paper as GPT-4, Claude 3, and Gemini 1.5, also report a majority of these illusion-illusions as illusions. Push a little on the prompt, prepend 'In the following visual illusion,' and the failure rate climbs further: nearly all illusion-illusions get flagged as illusions by Claude 3, GPT-4o, and Gemini Pro. Many control cases, simple images not even designed to resemble illusions, get called illusions too.

Why this matters if you are shipping a product on top of a vision model: the finding suggests these systems are often pattern-matching to surface features that co-occur with a category, rather than actually parsing what is in the frame. Any workflow that leans on a VLM to describe a chart, read a diagram, or verify a piece of visual evidence inherits that failure mode. A user prompt that hints at what the image 'should' be is enough to steer the answer.

The honest caveat is that this is a short, single-author study, nine pages and five figures, and the reported results are described as majorities and 'nearly all' rather than a full accuracy table you can drop into a slide. What the paper does not give you is a head-to-head score across the eight models it names, or a fix. The useful direction is the test itself: a cheap, adversarial 'there is no illusion here' probe is now on the shelf, and it belongs in the eval kit of anyone deploying a vision model where being confidently wrong is expensive.