AgentLens Scores Coding-Agent Trajectories, Not Just Pass/Fail

A new benchmark paper takes a small but sharp swing at how the industry ranks coding agents right now, and the argument is worth reading if you have ever tried to buy one of these tools on the strength of a leaderboard number.

The pitch, in an arXiv paper posted this week, is straightforward. Most code-agent benchmarks reduce a run to a single bit, did the task pass or not, but the people who actually use these agents experience the entire trajectory: how the agent follows instructions, uses its tools, verifies its own work, recovers from mistakes, and talks to them along the way. AgentLens is built to score that whole trajectory rather than just the endpoint.

The methodology pairs formal verification, where an objective check exists, with LLM-written trajectory reviews and side-by-side comparisons, so each run produces a readable explanation of why the score is what it is. The authors say they already use it internally to diagnose model behavior, compare successive versions of their own agent, and catch product regressions in a nightly evaluation pipeline. The benchmark is released as open source.

Why this matters if you are not building an agent yourself: procurement decisions and hype cycles both currently ride on single-number scores that even the vendors building these systems are quietly working around. A trajectory-aware evaluation makes it easier to tell an agent that solved a task cleanly from one that stumbled into the right answer, and that distinction gets more consequential once the agent is sitting inside a real engineering workflow.

The honest caveat is that the abstract does not include head-to-head numbers, does not name which agents scored where, and does not report how closely an LLM judge's trajectory review lines up with what a senior engineer would say about the same run. Trading one imperfect metric for another that is harder to audit is a real risk. Still, the direction is where the interesting evaluation work is going, and the code being open source means other groups can push on it rather than take the authors at their word.