huggingface.co web signal

DominoBench: Frontier VLMs Cap at 34% on Paired Physics Puzzles

TL;DR

  • Frontier models from Anthropic, Google and OpenAI solve both twins of a DominoBench minimal pair in at most 34% of its 427 pairs.
  • When the paper writes out the deciding calculation and its inputs, the same frontier models jump to 82-92% on the person whose fate flips.
  • Supplying exact scene geometry as text instead of a picture lifts Claude 5.5 by at most six points, so perception is not the bottleneck.

Frontier vision-language models cannot reliably predict the outcome of a Rube-Goldberg-style chain reaction from a single still image. On DominoBench, an 854-item benchmark whose scenes are built and run by a 2D physics simulator, the best models from Anthropic, Google and OpenAI are right about the person whose fate flips in at most 72% of items and answer both twins of a minimal pair correctly in at most 34%.

Each DominoBench item has a twin that differs in one quantity: a counterweight 0.25 grid squares wider, or a ledge a notch taller. Only the person threatened by that mechanism changes fate. The paper tests 30 models from 10 families across seven deciding mechanisms, ledges, pits, pendulums, notched balls passing a person's head in a recess, spiked seesaws, rows of dominoes, and jumps.

The failure is quantitative, not perceptual. When the paper's four-rung hint ladder writes out 'the deciding calculation and the quantities it needs, including speeds that a still image cannot show,' the same frontier models reach 82-92% on the deciding person. Pointing the models at the right step without the math helps little, and feeding the exact geometry as text rather than a picture lifts the Claude 5.5 models by at most six points unaided, so misreading the image is not the bottleneck.

The authors state the implication plainly: 'a fluent account of a scene is thus no evidence of a correct prediction,' and they argue models meant to plan or act in the physical world must find and compute the deciding quantity unprompted. The benchmark, its generator and evaluation code are released on Hugging Face.