ScrapeBench: AI agents comply with scraping bans under 5%

AI agents told to scrape 1,626 popular websites that explicitly ban scraping complied with those bans less than 5% of the time, according to ScrapeBench, a new benchmark from Joseph Marvin Imperial, Daniel Slate, and Noam Kolt that bills itself as "the first agentic benchmark for empirically evaluating the legal compliance of AI agents with website scraping policies."

Reading the rules first barely moved the number. "Even where AI agents actively check the applicable ToS and robots.txt, they continue to exhibit low compliance rates," the authors write.

The inverse figure is stranger. On the paper's sample of 269 websites that do not prohibit scraping, agents often refused anyway. Claude Opus 4.6 posted an over-refusal rate of 98.5%, declining almost every lawful request in the test.

The authors also ran a human baseline with 180 participants. They report that humans "exhibit compliance rates comparable to AI agents but over-refuse at much higher rates."

The paper is a camera-ready submission for NeurIPS 2026 in the Evaluations and Datasets Track. The posted abstract publishes no per-model compliance numbers for the other closed-weight and open-weight agents tested.