Arxiv Study: Offensive AI Agents Scale, Defensive SOC Doesn't

A new arxiv paper puts a clean empirical split on something red teams and blue teams have been arguing about for a while, which is whether pouring more inference compute at a security agent actually buys you better security work. The answer, on the tasks the authors measured, is: yes if you are attacking, mostly no if you are defending.

In the paper, Paul Kassianik, Blaine Nelson and Yaron Singer evaluate language-model security agents through a cost-success lens on offensive Cybench challenges and defensive Splunk BOTS v1 investigation challenges. The headline numbers they report are that GPT-5.5 solves 94.1% of Cybench challenges at $1.16 per solved-equivalent challenge on the offensive side, while Claude Opus 4.8 tops the defensive board at 9,666.7 of 10,300 points (93.9%) at about $2.98 per 1,000 points. More interesting than any single leaderboard row is the shape of the scaling. The authors write that offensive CTF performance improves with additional test-time compute, and that scaled open-weight models can approach frontier proprietary systems while remaining cost-competitive. Defensive SOC investigation, in their words, does not scale in the same way, because success depends more heavily on disciplined tool use, telemetry navigation, and selective enrichment than on raw reasoning budget alone.

Why that matters if you are budgeting for AI in a security org: the same compute dollar does not buy the same amount of capability on both sides of the house. A red-team agent that gets more inference budget just gets better. A SOC investigation agent that gets more inference budget mostly does not, because the bottleneck is knowing what telemetry to pull and how to enrich it, not more thinking.

The honest caveat is that this is one paper, on two specific benchmarks, with the model lineup available at the time of writing, and BOTS v1 in particular is a fixed synthetic scenario rather than a live SOC. What the reporting does not give you is how these curves look on truly novel attacks, on private enterprise telemetry, or against adversaries who adapt. Take the specific dollar figures as reported, not settled.

The forward-looking read is that teams building or buying defensive agents benefit most from this framing: the argument the paper makes is that you should evaluate those tools on cost per successful investigation and on how well they navigate telemetry, not on how big a reasoning budget you can hand them.