SWE-bench Science: top coding agent scores below 50% pass@1

Even the best coding agent tested clears fewer than half the tasks in a new scientific-code benchmark.

'SWE-bench Science: Can Coding Agents Resolve Engineering Tasks in Science?', posted to arxiv by Zhipeng Xu and colleagues, assembles 119 tasks from 98 GitHub repositories across 20 scientific domains, sorted into three task paradigms the authors label Issue-driven, Expert-exploratory, and Engineering-integration.

The topline: 'Even the best-performing agent, Claude Code with Opus-5 (max), achieves a pass@1 below 50%.' No per-domain scores appear in the abstract.

The framing motivating the work is that scientific code doubles as measurement apparatus. 'Software increasingly functions as part of the scientific instrument itself,' the authors write, 'making failures in scientific code capable of compromising not only program behavior but also the evidence underlying scientific conclusions.'

The failures cluster into four categories the authors name: 'deficits in scientific knowledge or abstraction, misguided exploration or surface-level repair, incomplete repair coverage or system integration, and failures to generalize scientific knowledge beyond observed cases.' Code and leaderboard live on the project's GitHub, which supports Codex, Claude Code, and mini-swe-agent as runtime harnesses.