Picture a price monitoring job that ran fine all summer and, sometime after 15 September 2026, started returning blocks, with nobody on the team changing anything.
One possible suspect is a changed default around Cloudflare Pay Per Crawl and its new sibling, Pay Per Use. According to Cloudflare's 1 July 2026 announcement, new domains with ads now block training crawlers and AI agents on ad pages by default, while search crawlers stay allowed. If your targets are long-established sites, the cause may be a publisher's own setting rather than the new default. Your logs will tell you which.
Most teams are not the target, but some will be collateral. Here is which side you are likely on, what it costs, and what you can legitimately do.
⏱ 15 min read Updated October 2026