BAAI AgentSWE Benchmark: Strongest Coding Builder Scores Just 19.1/100 Creating New Agents

Beijing Academy of Artificial Intelligence researchers released AgentSWE, a 25-task benchmark testing whether coding agents can build agents that generalize beyond visible development cases. The strongest builder averaged 19.1/100 on Creation, 31.6/100 on Editing, and closed less than 5% of the gap on Optimization — with runs scoring 90+ on development cases dropping to 9.9 on held-out cases. The team catalogs three recurring failure modes: overfitting, hollow runtimes that pass validity gates while producing nothing, and unsupported completion claims.