SWE-Bench ProMax caps top coding agent at 41.2% resolve rate

A new evaluation published for COLM 2026 sets out to fix two problems that keep eating coding-agent benchmarks: they saturate, and their tests are broken. SWE-Bench ProMax is 170 refactoring tasks drawn from real GitHub commits across seven languages (Python, Java, TypeScript, Go, C, C++ and Rust), with every issue description rewritten from scratch and every test suite hand-reviewed. The authors cite a prior audit finding that nearly 60% of unsolved SWE-bench Verified instances had flawed tests, split between 35.5% overly narrow and 18.8% overly broad; OpenAI has since deprecated SWE-bench Verified entirely.

The scale change is the point. Gold patches in ProMax average 11.4 source files and 261.6 lines of code per instance, and 30% of tasks touch more than ten files. On SWE-bench Verified, by contrast, 86% of instances modify only a single file. A representative ProMax task migrates NASA's F'Prime flight software across 244 files while preserving identical runtime behavior.

Six frontier models were tested under two agent scaffolds, mini-swe-agent and OpenHands, with a 300-step and $10-per-instance cap. The best result is GPT-5.2 at 41.2% resolve rate on OpenHands, well below the 75%+ that top agents now post on SWE-bench Verified. Claude Sonnet 4.6 lands at 38.8% but costs $4.77 per instance, roughly one-twentieth of which is what Zhipu's open-weight GLM-5 spends to reach a comparable 36.5% (at $0.24). Qwen3.5 also hits 36.5%, at $0.78.

The failure-mode analysis is arguably more useful than the leaderboard. Agents track the gold patch on small changes but stall on large ones: their file-count distributions plateau around ten files while the gold patch curve only reaches ninety percent near twenty. Failed runs also consume more interaction rounds than successful ones, what the authors call unproductive exploration: agents reading, editing, hitting a test failure, reverting, and never expanding scope.

A few things the paper does not close out. Coverage skew is severe (Angular alone contributes 25 of the 28 TypeScript instances), so per-language rankings likely reflect the idioms of a specific codebase as much as any general language effect. And the analysis stops at Pass@1, so there is no measurement of how much a second attempt would rescue on the failing runs. If you are budgeting for a code-agent pilot, this is the first benchmark in a while that gives you a defensible reason to test the cheap open-weight options against the frontier before you sign a Sonnet-sized invoice. Evo-Bench landed the same week if you want a second read on where the ceiling sits.