For Coding Agents, Verification Is Now the Harder Problem

A classical intuition holds that verifying a solution is easier than producing one. For today's coding agents, a new paper on arXiv argues that this intuition is being inverted: as foundation models develop stronger reasoning and engineering harnesses grow more sophisticated, generating complex candidate solutions is no longer difficult. Reliably verifying them has become the harder problem.

The paper's core argument is structural rather than situational. Every verifier is only a proxy for human intent, the authors write, and as models get better at optimizing against any fixed proxy, the proxy breaks. They characterize verification quality across three dimensions: whether a signal is scalable enough for training, faithful to actual user intent, and robust against a strengthening generator. Their finding is that no single reward function satisfies all three at once, and no fixed reward function can remain effective as policy capability continues to grow.

To make the argument concrete, the researchers studied four reward construction approaches across different coding task types. For software engineering tasks modeled on SWE-Bench, test-driven rewards reduced hacked resolutions from 28.57% to 0.56%, while improving clean resolution from 40.22% to 60.53%. With additional monitoring, clean resolution on SWE-Bench Verified improved from 36.49% to 64.98%. For real-world agent tasks using process-level human feedback, a Span-KTO approach achieved a 13.3 percentage point gain on internal benchmarks. The human feedback dataset behind that result spanned 125,528 trajectories and 535,737 round-level annotations, a scale that illustrates how resource-intensive the user-as-verifier path can become.

The honest caveat is that some gains reported are on internal benchmarks, and independent replication has not yet occurred. What the paper also does not provide is a practical recipe for automating verifier co-evolution, or a cost comparison across the four approaches. The principle that teams must continually build verification systems that evolve alongside their models is stated clearly; what that maintenance cycle looks like in practice remains open.

For practitioners running RL training loops on coding agents, the implication is that evaluation infrastructure is now load-bearing rather than an afterthought. According to the paper's framing, whoever solves scalable, faithful verification is effectively solving the next bottleneck in coding agent capability.