Found first: a primary source the press has not covered yet.
A new benchmark called Co-Design Bench tests coding agents on 273 tasks drawn from real user sessions and finds the strongest of ten evaluated model endpoints satisfies an average of 69.9% of acceptance criteria per task but fully completes only 26.4% of tasks. The paper, "Co-Design Bench: Multimodal Collaborative Coding Evaluation for Agentic Models", was posted to arXiv on October 26, 2026, by Xilin Wei, Shengyuan Ding, and fourteen co-authors.
What the source says
The 273 benchmark tasks were curated from 37,463 deduplicated queries drawn from 26,314 real user coding sessions, with 2,331 expert-authored acceptance criteria written to judge completions. The evaluation covers two tracks: Multimodal Editing, for localized edits with optional multimodal inputs, and Repository Editing, for coordinated repository revisions. Ten model endpoints were evaluated. The strongest model averaged 69.9% of criteria satisfied per task, yet fully satisfied all criteria in only 26.4% of tasks. Performance falls further on edits spanning three or four change types compared to those spanning one or two.
Why it matters
The gap between 69.9% average criteria pass rate and 26.4% full-task completion rate is 43.5 percentage points. Partial-credit scoring, which most coding benchmarks use, lets a model accumulate a high aggregate score even when it cannot satisfy every requirement in a single task. Because Co-Design Bench derives tasks from actual user sessions rather than synthetic prompts, its failure distribution reflects work developers genuinely submit. The additional drop-off at three and four change types suggests the reliability problem compounds as task complexity grows, which is precisely where real-world coding requests tend to cluster.