Mastermind hits 84.5% on CyberGym repo-scale CVE reproduction

Something interesting from a fresh preprint that reframes where the useful work is happening in LLM security agents. A paper called "Mastermind: Strategy-grounded Learning for Repository-Scale Vulnerability Reproduction" reports an 84.5% pass rate on CyberGym, a benchmark that has agents inspect real codebases, infer input grammars, build proof-of-concept exploits, and verify fixes. The stronger baselines it compares against, open-book PoC context at 60.0%, best-of-8 sampling at 63.0%, and iterative improvement at 77.0%, are respectable, so the jump is not against a strawman.

The interesting part is the argument behind it. The authors claim "strategy, rather than the full action trajectory, is the right learning unit" for these agents, because strategy is compact enough to optimize, concrete enough to guide execution, and stable enough to store and reuse across attempts. Concretely, they train a planner using SFT plus a milestone-based GRPO variant, and leave the executor frozen. That framing matters because most agent research so far has focused on training the whole trajectory, which is expensive and executor-specific.

Where the claim earns its keep is the transfer result. The same trained planner reportedly lifts a frozen GPT-4 mini from 45.0% to 60.0%, and GLM-4 from 58.5% to 71.0%. If that holds up on independent inspection, it says a small piece of training can meaningfully improve a range of frozen executors on the same task, which is a very different economic picture than fine-tuning the whole model per capability.

The honest caveat is that this is a preprint on one benchmark, evaluated on 200 held-out tasks drawn from the same distribution as its 260 training tasks. Real-world CVE diversity is much wider, and repo-scale vulnerability reproduction cuts both ways, since the same capability that helps defenders reproduce and patch faster also helps anyone assembling an exploit corpus. What the arxiv abstract does not answer is whether the planner weights or the CyberGym task set will be released, and how much of the lift survives on repos and vulnerability classes not represented in training.

For teams building security agents on frozen APIs, the direction to watch is not a better executor but a trainable planner sitting over any executor. That path is materially cheaper, and if the transfer story generalizes, it is where the next round of practical gains for defensive tooling is going to come from.