OpenAI drops SWE-Bench Verified, backs Pro after flaw audit

A widely competed coding benchmark just got yanked out from under the leaderboard race. OpenAI has stopped reporting scores on SWE-Bench Verified, the 500-task Python evaluation that most frontier labs had been publishing against, and is asking the industry to move to Scale AI's SWE-Bench Pro instead, according to the OpenAI post and reporting from The Decoder.

The reasoning is unusually direct. OpenAI's evals team audited 138 of the hardest problems in Verified, about 27.6% of the dataset, and found that 59.4% of them had material issues in test design or problem descriptions. Roughly 35.5% enforced specific implementation details unrelated to the actual task, and 18.8% checked for functionality the problem description didn't ask for. Layered on top of that, OpenAI reports that GPT-5.2, Claude Opus 4.5 and Gemini 3 Flash Preview could reproduce some original fixes from memory, which is the polite way of saying the benchmark has leaked into training data.

Why this matters if you buy or ship coding assistants: the numbers on the marketing pages have quietly stopped meaning what they used to mean. On Verified, top models had been parked around 80%. On Pro, the same models drop by roughly 35 points on the same kind of task. If your procurement checklist reads "must score above X on SWE-Bench," X now refers to a different benchmark with a very different ceiling, and the switch changes who looks best.

The honest caveat is that Pro isn't a settled answer either. A May 2026 audit by Datacurve reported Pro's own graders mis-graded roughly a third of trials, accepting wrong patches about 8.5% of the time and rejecting correct ones about 24%, though those figures haven't been confirmed by Scale AI. What the reporting doesn't yet give you is whether Anthropic, Google DeepMind and Meta will actually standardize on Pro's public split, or quietly reach for private in-house evals as the credible signal instead.

The people who benefit here are smaller model builders and open-source teams who could not out-memorize a leaked benchmark, plus buyers who now have a more honest ceiling to compare against. The people exposed are labs whose headline coding-agent numbers were doing more work in the sales deck than in production.