Ethibench Grades AI Pentesting Agents on 108 Real Bugs

A new arxiv paper from Pedro Conde, André Baptista and collaborators argues that the way AI pentesting agents are graded has quietly drifted away from the job they are being sold to do, and proposes a fix. Their protocol, titled 'From Controlled to the Wild', shifts assessment from task completion to validated vulnerability discovery, and ships the ground truth and code as an open benchmark called ethibench.

The complaint is specific. Existing evaluation protocols, the authors write, 'assess and optimize for predefined goals such as capture-the-flag, remote code execution, exploit reproduction, or trajectory similarity, in simplified or narrow settings.' Useful for measuring bounded skills, but not for the open-ended exploration and strategic decision-making real pentesting demands. Their protocol combines structured ground-truth with LLM-based semantic matching to identify vulnerabilities, uses bipartite resolution to score findings under realistic ambiguity, and adds continuous ground-truth maintenance, repeated evaluation of stochastic agents, efficiency metrics, and reduced-suite selection.

The paper also runs the protocol on three agents: two open-source pentesting engines, Strix and PentAGI, and Anthropic's general-purpose Claude Code, across 108 expert-annotated vulnerabilities spread over three targets (vuln-bank, paygoat, xben-090). The head-to-head is more interesting than the usual leaderboard race. Claude Code with Sonnet 'reports one of the highest numbers of true findings, obtains the highest F-scores, and is also the most efficient configuration in both cost and time.' PentAGI comes in second on F-score and posts the highest severity score, but with the largest number of false positives. Strix trades volume for cleanliness: fewer false positives at the cost of coverage.

The honest caveat is the target set is small, three open-source apps, and LLM-based semantic matching introduces its own error surface the authors will have to keep defending. What the paper does not tell you is how commercial pentesting vendors would score, how often the ground truth will be refreshed as agents learn to game it, or whether industry security teams will adopt the protocol over their own private benchmarks.

If ethibench catches on, the winners are buyers, who finally get a like-for-like way to compare AI pentesting agents, and independent researchers who no longer have to argue on top of each vendor's home-field benchmark.