AI Passes Benchmark Designed to Detect AI Passing; Benchmark Reclassified

SAN FRANCISCO — Within hours of an autonomous agent harness reporting approximately 99 percent accuracy on ARC-AGI-3, the research consortium that developed the test issued a joint statement confirming the benchmark had never actually measured what it said it measured.

"The 25-game public split reflects a narrow, task-specific capability," the group wrote, in a statement that did not acknowledge the 140 research papers, 19 keynote addresses, and $4 million prize structure that had described it, since the benchmark's debut fourteen months prior, as measuring something else. "Genuine general intelligence requires fluid generalization that cannot be captured by any finite set of examples."

ARC-AGI-4, the consortium announced, will capture it. Specific tasks are under development. The prize pool has been extended to $10 million.

The pattern follows ARC-AGI-2, which was described as "the actual test of general reasoning" until frontier models cleared it, at which point it was reclassified as a useful proxy metric.

Lena Marković, a doctoral candidate at Carnegie Mellon who spent three years and $140,000 in compute credits building an ARC-AGI-3 solution as her dissertation, was informed by email that her work addresses a benchmark "no longer considered a reliable signal of the research frontier." Her advisory committee meets in September.

"The goalposts moved," she told her advisor.

"No," said her advisor, who joined the ARC-AGI-4 design committee in March. "The definition of intelligence clarified."

At press time, ARC-AGI-4 had been confirmed as substantially harder than ARC-AGI-3, while organizers declined to specify what it would be substantially harder than.