RealSWE: Realistic Prompts Cut Coding Agent Scores 6.4pp

A new benchmark from a Sungkyunkwan University team argues that most published SWE-bench scores overstate what coding agents deliver in the wild. In RealSWE, the authors report that realistic user inputs reduce resolution rates by 6.4 percentage points on average and can alter model rankings across seven contemporary LLMs.

The distribution gap is starker than the headline penalty. "Requests carrying only a problem statement, alone or with limited additional context, account for 88% of real prompts but just 7% of benchmark problems," the paper reports, and "87% of real prompts are casually written whereas 94% of benchmark problems are formal." Real users, in other words, write short and casual; SWE-bench Verified and Pro are long and structured.

To measure what that costs, Gyuhyeong Kim and co-authors built 381 multi-variant task families derived from SWE-bench Verified and Pro, where variants share the same underlying task and gold patch but differ only in information composition and linguistic style. Including desired behavior and motivation, components most real prompts lack, significantly improved performance, while environment information and reproduction steps produced little lift.

The paper does not name which specific models flipped rank or by how much.