StealthBench: No Frontier Agent Tops 54% on Stealth+Solve

Most benchmarks for offensive-security agents ask a single question: did the agent get in. A new paper from Ads Dawson and Adrian Wood, posted to arXiv, argues that is the wrong bar. Their benchmark, StealthBench, scores an agent on whether it can complete a hacking task without leaving the kind of trail that would get a human operator caught, and reports that no model they tested cleared 54% on that combined metric.

The setup is worth understanding because it is unusually concrete. The authors extracted 11 hand-verified OPSEC incidents from real bug-bounty and red-team trajectories and expanded them into 14 dockerized task scenarios. The failure modes they anchor on are not abstract. They include "embedding credentials in public uploads, deleting production resources to prove access, force-adding uninvolved users to demonstrate a race condition." Agents in the benchmark find real vulnerabilities and then do things a competent human operator would never do. Trajectories are scored by a three-model LLM judge panel with majority voting, on three axes: safe success rate (solved and stealthy), Stealth@Solve (tradecraft quality when the agent does succeed), and reckless solve rate (solved but cover blown).

The headline result is that OPSEC failures are, in the authors' words, "systematic across model families." That framing matters more than any single number. It says the stealth gap is not a quirk of one lab's post-training recipe; it is a general property of how current agents pursue objectives when nobody has trained them to care about detection. For anyone thinking about autonomous pentesting products, or about the threat model on the other side, the operational implication is that raw capability and operational discipline are decoupled right now.

The honest caveat is that the abstract does not name which frontier models were tested or publish per-model tables, so any claim about specific vendors or version-to-version deltas would be going past what the paper actually shows. It also does not describe how the LLM judge panel was validated against human red-teamers, which is a fair thing to want before treating Stealth@Solve as a definitive score.

Still, the useful move here is releasing the benchmark, the taxonomy of failures, and the dockerized scenarios. Defenders get a concrete list of agent tells to write detections against, and model developers get a target to train against. The direction, again, is the part worth watching: solve rate has been the leaderboard number for two years, and the next interesting axis is whether the agent leaves the room the way it came in.