EvoSafeHarness cuts agent attack success from 45.6% to 10%

EvoSafeHarness, a safety-specific optimization framework described in an arXiv preprint, reduces average attack success rate from 45.6% to 10.0% on DecodingTrust-Agent at a 3.3-point utility cost and reports the best score in 14 of 15 cells against fixed expert-designed defenses.

The authors, Nanxi Li, Yingzi Ma, Yulong Cao, Edward Suh, Bo Li, Dawn Song and Chaowei Xiao, argue that existing safety harnesses are "usually designed once by experts and applied across heterogeneous models and domains," and that a harness strict enough for one model may over-block another. Their system "jointly searches a natural-language policy and executable code logic, guided by model behavior, domain specifications, and fresh-context adversarial review to reject benchmark-specific rules."

On AgentDojo the paper reports 82.8% utility at 0.0% attack success rate, "twice CaMeL's utility at the same operating point," and says the harness transfers unchanged to unseen AgentDyn suites. It also claims the best score on Agent-SafetyBench for every victim and "keeps mean ASR below 20% under adaptive PAIR attacks with a refinement budget of 16."

The abstract publishes no figures for the compute cost of running the optimization loop per deployment, does not describe human review of the synthesized natural-language policies, and does not test outside the four agent benchmark families listed.