Ai2's StateBench Drops Frontier Models to 13% on State Tracking

Frontier language models can handle variables whose values never change. Ask them to keep up when those variables start pointing at each other, and they collapse. In a blog post dated November 3, the Allen Institute for AI released StateBench, a benchmark that walks models up four rungs of difficulty: immutable values, reassignment by value, self-referential updates, and finally assignment-by-reference, which Ai2 calls "general finite" problems. On that last rung, Gemini 3.8 Flash, Claude Opus 5.5, and GPT-6 Sol and Luna held up on short inputs. On long ones, Ai2 reports, "accuracy fell sharply as inputs grew longer, dropping to 13% or lower on the longest general finite problems."

The sharper result sits on the architecture side. When models were trained directly on the task, Gated DeltaNet and hybrid variants "generalized almost perfectly on r-trivial, aperiodic, and soluble problems out to the longest sequences we tested, tens of thousands of assignments," the team writes, and stayed close to perfect at roughly ten times the training length. Transformers degraded on longer sequences, and in Ai2's words, "never learned general finite problems" at all, regardless of training.

It lands alongside an arXiv compaction paper moving through our long-context coverage the same week, which argues you can summarize your way out of needing most of the window. StateBench points at the other side of the question: whether the model can hold the state at all. "We hope StateBench gives researchers a finer-grained way to find where models break down," Ai2 writes.