paper web signal

Coding Agents Solve Early, Stop Late on SWE-bench: Paper

TL;DR

  • A new arXiv study finds 38--82% of successful coding-agent runs reach a correct fix on their very first source-code edit.
  • Post-solution work runs a median 35--37% of steps for Qwen3-Coder and 54--60% for Laguna-XS.2 on SWE-bench Verified and Multilingual.
  • In 1--8% of runs the agent's extra work undoes an earlier correct patch, leaving the final submission incorrect.

Coding agents arrive at the correct fix well before they stop running. A new arXiv paper, "Termination Lag: Coding Agents Solve Early but Stop Late," reports that "38--82% of successful runs achieve a correct solution on their initial source-code edit."

Authors Musa Cim, Burak Topcu and Mahmut Taylan Kandemir characterize the pattern across five models, two agent frameworks and three SWE-bench datasets, recording every distinct patch produced during execution. Post-solution steps account for a "median of 35--37% for Qwen3-Coder and 54--60% for Laguna-XS.2 in successful runs on SWE-bench Verified and Multilingual." "Agents spend most of these additional steps verifying their work and exploring the repository," they write.

Sometimes the extra work costs the whole run: "In a small fraction of cases (1--8%), this continued activity leads agents to undo their earlier correct patch, leaving the final solution incorrect."