Third Intelligence WARC learns 56 of 69 ARC-AGI-3 mechanics
TL;DR
- WARC is a 35M-parameter vision transformer that updates all of its weights after every transition in a single play of a new game.
- On MechEval, WARC reaches 80% accuracy on 56 of 69 mechanics, with half crossing the threshold inside about 80 exposures.
- Paired with a graph explorer, WARC-Agent triples the official efficiency score from 0.15 to 0.46 on 25 public games and scores 0.20 on the Kaggle hidden leaderboard.
Third Intelligence's WARC, a 35M-parameter vision transformer that rewrites all of its weights during a single play of a new game, reaches at least 80% prediction accuracy on 56 of 69 mechanics catalogued in a companion benchmark called MechEval, with half of those mechanics crossing the threshold inside roughly 80 observations.
The paper on Hugging Face by Haruto Kawano, Shota Takashiro and Kohei Hayashi (Third Intelligence, Inc. and the University of Tokyo) targets ARC-AGI-3, a benchmark of interactive grid-world games defined by their programs. MechEval labels 69 recurring mechanics across 20 of the public games. With full-parameter test-time training, 20 gradient steps after every valid transition, WARC hits 69.1% mechanic accuracy across all exposures, against 18.4% for the same model with weights frozen, 32.9% for an episodic-retrieval baseline, and 30.1% for a 50M-parameter DreamerV3. The abstract is blunt about the frozen case: 'without updates only one in seven does at all.'
Paired with a fixed graph explorer that hashes states and picks actions whose predicted frames differ most from frames already seen, WARC-Agent 'triples the explorer's official efficiency score from 0.15 to 0.46, mostly through test-time training' on the 25 public games. On the hidden games of the Kaggle leaderboard the entry scores 0.20, well short of the 1.39 Oracle ceiling the paper reports.
The ablation sharpens the claim. Strip the pretraining on other games and mechanic accuracy drops 34.1 points; update every five transitions instead of every one and it drops 10.5. Compute is not free either: roughly 0.74 seconds per action on an RTX PRO 6000 added up to 8.6 hours for the full Kaggle submission.
It lands as one of 51 research papers tracked in our research feed over the past 90 days, notable less for its headline leaderboard number than for showing a small model, aggressively updated at test time, outperforming much larger frozen baselines on the exact axis ARC-AGI-3 was built to probe.
Originally reported by huggingface.co
Read the original article →Original headline: WARC World Model Learns 56 of 69 ARC-AGI-3 Game Mechanics in ~80 Exposures With 35M Params