Apple paper: reasoning models collapse past complexity threshold

Apple machine-learning researchers report that leading reasoning models, including Claude 3.7 Sonnet in thinking mode, DeepSeek-R1, and OpenAI's o3-mini, hit "a complete accuracy collapse beyond certain complexities" once puzzles are scaled past a threshold, even with token budget still available. The paper, The Illusion of Thinking by Parshin Shojaee, Iman Mirzadeh, Samy Bengio, Mehrdad Farajtabar and colleagues, tests reasoning-tuned LLMs on controllable puzzle environments (Tower of Hanoi, River Crossing, Blocks World) designed to avoid contamination from standard math and coding benchmarks.

The authors identify three regimes. On easy puzzles, standard LLMs match or beat the reasoning variants. In the middle band, chain-of-thought helps. Past a complexity ceiling, both collapse to zero. The paper also flags a counterintuitive scaling limit: "their reasoning effort increases with problem complexity up to a point, then declines despite having remaining token budget."

The result cuts against the industry narrative that longer thinking traces reliably unlock harder problems, and it drew a fast rebuttal titled "The Illusion of the Illusion of Thinking," co-authored by researcher Alex Lawsen with Anthropic's Claude Opus 4, arguing that the observed collapse partly reflects the test setup rather than a true reasoning limit. Two of the AI experts we track shared the paper this week.