r/AI_Agents: 3 Months, 11 Million Crawler Events Across 34 Sites — GPTBot Ignores robots.txt Around the Clock While Other AI Agents Behave Radically Differently

A developer sharing 11 million real crawler-log events from 34 production websites over three months finds that AI web agents behave in dramatically divergent ways: GPTBot crawls relentlessly around the clock with minimal respect for robots.txt, while other AI crawlers show more conservative and variable patterns in timing, frequency, and policy compliance. The data carries practical infrastructure implications for teams whose sites are increasingly consumed by AI agents rather than human visitors. The thread is generating active discussion among builders comparing their own observations of which agents respect rate limits and crawl directives.