AI scraper bots swarm university libraries, blocking users

The University of North Carolina at Chapel Hill's library catalogue got more than 11,000 searches for Finnish music in a single day, coming from thousands of different internet addresses, CBC News reports. The same system was used to handling about 100 simultaneous searches during finals week; in December 2024 it was seeing 500 or more at a time, and the servers started locking out students, faculty and staff.

"They're just out there on the internet, just bombarding us with trying to grab up as much material as possible," Jill Strykowski, cataloguing and government documents lead at San Jose State University, told CBC. She said the scraping is causing "an incredible amount of work and headache."

The bots ignore robots.txt, request thousands of documents a second, and download the same material in every available format. Some test library card numbers and check out as much content as the account will hold. "Sometimes the bot traffic gets so bad that it causes the sites to go down completely for a little while," said Dean Seeman of the University of Victoria libraries.

The scale is not one campus. A 2025 GLAM-E Lab survey of 43 people maintaining online cultural collections found 39 had seen traffic rise over the previous year, and 27 of those blamed AI training data bots directly. Michael Weinberg, co-director of GLAM-E (a joint project of the University of Exeter and NYU Law's Engelberg Center), said the swarms are creating "a disincentive for libraries and other institutions to make parts of their collection available online."

Anthropic, OpenAI, Google and Microsoft have each said they support a standardized robots.txt opt-out signal that sites could use to block AI crawlers. Compliance is voluntary, and the agreement does not cover smaller AI startups or third-party scrapers.

Eric Hellman, executive director of Project Gutenberg, called the worst scrapers poorly built "monstrosities" and said "everything's gotten worse" since he first wrote about them.