Co-Design Bench: top coding agent completes just 26.4% of real-user tasks

The strongest of ten coding-agent endpoints tested on Co-Design Bench passes 69.9% of per-task acceptance criteria on real-user sessions but fully satisfies only 26.4% of tasks, which the paper frames as a testbed "for studying the gap between partial requirement satisfaction and complete task success under agentic execution for real-user application revisions."

The benchmark is drawn from live traffic rather than synthetic prompts: 273 tasks curated from 37,463 deduplicated queries across 26,314 real user coding sessions, scored against 2,331 expert-authored acceptance criteria hidden from the agent. It splits into two tracks, Multimodal Editing and Repository Editing, covering localized edits with optional image, audio, video or document inputs alongside coordinated repository revisions.

The headline comparison is the gap itself. Averaging per-criterion passes tops out at 69.9% for the strongest endpoint; whole-task completion, meaning every criterion for a task satisfied together, stops at 26.4%. The paper also reports "lower performance on repository edits spanning three or four change types versus one or two," without publishing per-track or per-model numbers in the abstract.