OpenFrontiers: LLMs solve 7 of 40 problems without the web
TL;DR
- OpenFrontiers tested seven LLM agents on 40 resolved math problems without web access; the best scored at most 7 of 40.
- The benchmark pairs a 12,151-problem corpus with a 200-problem retrospective set split between 110 resolved in 2026 and 90 still open.
- Live-web performance, the paper reports, mostly reflects retrieval of published solutions rather than independent rediscovery.
Seven large language model agents, tested on 40 resolved research problems in mathematics without web access, solved at most 7 of them. That is the headline number from OpenFrontiers, a benchmark paper by Jiarui Yao, Ruida Wang, and Tong Zhang that sets out to separate scientific rediscovery from retrieval.
The authors built a cross-disciplinary corpus of 12,151 open research problems alongside a 200-problem retrospective benchmark: 110 problems resolved in 2026 and 90 still open, each with a Lean 4 statement checked by automated probes and two independent reviews. From that, a 50-problem HQ-50 subset was carved out by quality score.
Proofs are graded by "two independent checkers in an isolated sandbox under closed-book, frozen-retrieval, and live-web conditions." The closed-book result, at most 7 of 40, is the paper's cleanest number. The live-web picture is messier: the abstract reports that "live-web success mostly reflects retrieval of published solutions."
The abstract publishes no per-agent accuracy numbers, no breakdown by condition beyond those two anchors, and no list of which seven agents were evaluated. The 110 problems resolved in 2026 also raise a training-cutoff question the abstract does not address.
Originally reported by paper
Read the original article →Original headline: OpenFrontiers: LLMs Solve At Most 7 of 40 Open Research Problems Without Web Access — Web Gains Are Mostly Retrieval