Cloudflare will block AI training bots on ad pages Sept 15

One thing worth watching in the AI-versus-publishers slow burn: Cloudflare has picked a date. On September 15, 2026, the company announced that new defaults will kick in for how AI crawlers are treated on sites behind its network. The old binary of block-AI-bots-or-don't is gone; in its place is a three-way split between Search, Training and Agent traffic.

The mechanics are the important part. Cloudflare defines Search as behavior that collects or indexes content to answer questions later, Agent as automated behavior acting in real time on a person's behalf, and Training as a crawler taking content to train or fine-tune a model. Under the new defaults, Training and Agent will be blocked on pages that display ads, while Search will remain allowed. The rules apply to all new domains onboarding to Cloudflare, new customers, and existing customers on the Free tier, with an opt-out window before September 15.

The framing from CEO Matthew Prince is the tell. "Now that the majority of traffic on the Internet is non-human, we must go further and act faster so that a sustainable ecosystem can emerge," he said, according to The Register. The pitch is that an ad is a signal the site owner meant a human to land there, so scraping that page to train a model or to feed an agentic answer is a different transaction from search indexing.

Two things ship alongside the deadline. BotBase, a database of known bots with their verified status and declared purpose, is a view Cloudflare says it hasn't shown dynamically on the dashboard before. And Pay Per Crawl is being rebranded to Pay Per Use, with launch partners Ceramic.ai paying publishers when content appears in search results and You.com paying when premium content is accessed by an agent.

The honest caveats. The reporting doesn't spell out how mixed-use crawlers from Google, Apple and Microsoft will actually behave when Training is blocked and their bots fall under the most restrictive rule, and it doesn't give you a rate card for what Pay Per Use will pay a small publisher per use. The Search / Training / Agent categories are also declarations by the AI companies themselves, so anyone tempted to label a training run as 'Search' isn't stopped by a fingerprint in the announced policy.

The opening this creates is for publishers who sat out earlier AI content deals because the terms were binary. If Pay Per Use pays anything meaningful and Ceramic.ai and You.com are the first two willing to sign, the interesting question is who is second.