JetBrains: SWE-Bench Pro 93.8% Implementation, Real Devs Diverge

"new features (54.7%) and bug fixes (39.1%) account for 93.8% of tasks" on SWE-Bench Pro. In JetBrains' 4,782 real IDE sessions, the same two categories land at 25.9% and 18.9%.

That is the finding of "Coding-Agent Benchmarks Should Match Their Users' Task Flows", posted to arXiv by JetBrains researchers Igor Slinko, Yaroslav Golubev and Sergey Titov. The dataset the authors call Production Sessions covers roughly 14,000 user messages. Their analysis restricts to sessions with at least three user messages, which is 33% of the sample.

What the other time goes to: explain requests at 6.1%, general programming questions at 9.7%, plus planning, review, refactoring and execution help categories the authors say are largely absent from issue-derived benchmarks. Users don't stay in one lane either. The paper notes that "users switch between types throughout a session," and that long-session samples from three public interaction corpora show markedly different Task Flows.

On 700 SWE-Bench Pro tasks, the authors report that "solving the task sequentially in several steps approximately doubles agent cost without a stable change in resolve rate: the interaction protocol itself is an important dimension of evaluation."

Their proposed fix is SWE-TaskFlow, which converts verified SWE-Bench Pro tasks into multi-turn trajectories using prompt splitting and verifiable repository QA, scored with a TaskFlow Alignment Score (TFAS). The authors do not claim one interaction distribution is universally realistic; they argue "benchmarks should name a target use case and calibrate to measurements from it."

The paper does not break out the remaining task-type percentages, name which model powered the 700-task pilot, or compare resolve rates across vendors.