Alibaba SecRespond: no LLM fully solves any incident range

The interesting gap in the current generation of security-focused AI agents is not what they do when an alert fires. It is what they do when it does not. Alibaba-NLP's new SecRespond benchmark, reportedly the first evaluation designed for LLM agents doing post-compromise incident response, ran 23 frontier models through 10 cyber ranges built from real compromised cloud hosts. Not one of them fully solved a single range.

The setup is worth understanding, because it is a real workload rather than a capture-the-flag toy. Each range hands the agent a forensic disk snapshot of a compromised host together with the alerts, vulnerability scans, and baseline checks the host produced, and asks it to write a forensic report and a remediation plan. The ranges span 4 entry-point types, 21 ATT&CK techniques, and 5 operating systems, which is enough surface area to make the 'model got lucky on a familiar shape' excuse fairly weak.

The pattern the authors describe is consistent and, if you have watched actual SOC analysts work, unsurprising. Agents are reasonably good at running down leads that an alert has already put in front of them. They are meaningfully worse at pushing into the disk on their own to find silent intrusions the sensors missed, and at producing remediation plans that are both comprehensive and verified. The abstract's own wording is that models 'struggle to proactively investigate the disk for silent intrusions and to produce comprehensive, verified remediation plans, with no model achieving complete detection and remediation on any single range.'

The honest caveat is that the public abstract does not tell you which of the 23 models did best or worst, what the pass/fail bar is, or how the benchmark scores partial remediation. It is a cloud-host benchmark, so identity-plane and network-only incidents are outside its scope. And 'no model solved a single range' is a strong claim that will want independent replication before anyone quotes it in a board deck.

For teams building or buying autonomous SOC agents, the useful piece is that there is now a concrete, open target for the specific capability the current generation lacks. The dataset is on Hugging Face and the code is on GitHub, which means the next round of agents can be trained and evaluated against exactly the failure mode the paper names, rather than the alert-driven triage they were already reasonable at.