Red heart image nudges VLMs toward recursive code, paper finds

A red heart in a prompt can push a vision-language model toward a recursive answer on a coding problem that has nothing to do with hearts or recursion. That is the opening claim of a preliminary paper posted to arxiv on 27 October by Alexa R. Tartaglini, Zhengxuan Wu, Thaddäus Wiedemer, Martin Wattenberg, Been Kim and Yuxuan Li.

The surprising part is not that models can be steered, but that the steering happens behind the model's own back. The effect is "largely implicit," the authors write, because "models' direct interpretations and thinking traces about visual symbols diverge substantially from their observed behavior." Ask the model what the heart means and it will tell you one thing; watch what the heart does to its code and it does another.

When VLMs communicate visually with each other, the paper describes a gap between what senders put out and what receivers pick up: models tend to send shape primitives that feel intuitive both to humans and to the senders themselves, but those cues often fail to steer other models as intended. In further games using visuals to transmit abstract concepts and short stories, models get creative, "although some communications remain non-intuitive to humans."

The authors frame the results as preliminary and note that VLMs' "interpretation and production of visual meaning diverges from human conventions in surprising ways." Three researchers we track in our Who's Who directory posted the paper the same day.