Picture a price monitoring job that ran fine all summer and, sometime after 15 September 2026, started returning blocks, with nobody on the team changing anything.
One possible suspect is a changed default around Cloudflare Pay Per Crawl and its new sibling, Pay Per Use. According to Cloudflare's 1 July 2026 announcement, new domains with ads now block training crawlers and AI agents on ad pages by default, while search crawlers stay allowed. If your targets are long-established sites, the cause may be a publisher's own setting rather than the new default. Your logs will tell you which.
Most teams are not the target, but some will be collateral. Here is which side you are likely on, what it costs, and what you can legitimately do.
⏱ 15 min read Updated October 2026
Key takeaways
- Since 15 September 2026, new Cloudflare domains with ads block training crawlers and AI agents on ad pages by default. Search crawlers stay allowed.
- Pay Per Crawl charges for access. Pay Per Use, in beta since 30 September 2026, pays for what happens next. They are complementary, not a rename.
- Your crawler's category (Search, Agent or Training) and its declared identity decide whether you are allowed, charged, or blocked.
- Check your logs for 402 and 403 responses, declare your bot's identity, and compare blocked access with paid access before you budget.
What Changed: From Opt-Out to Ask-First
For years, access to a public page was something you took unless the site said no. Now Cloudflare blocks AI crawlers by default on new sites, and AI access is something you ask for.
It is a default flip, not a mature licensing market. There are no public figures on revenue or adoption. What Cloudflare does publish is traffic data: 52% of crawler requests were for AI training as of June 2026, up from 22% in spring 2025.
The timeline
- 1 July 2025: Cloudflare launches Pay Per Crawl as a private beta, built on HTTP 402 Payment Required.
- 1 July 2026: Cloudflare splits AI traffic into Search, Agent, and Training, each with its own setting.
- 15 September 2026: new defaults take effect. Training crawlers and AI agents are blocked on ad pages of new domains. Search stays allowed.
- 30 September 2026: Pay Per Use and the Monetization Gateway enter beta.
The two models are complementary, not a rename. In Cloudflare's own words: "Pay Per Crawl... charges for access. Pay Per Use pays for what happens next. Publishers can choose the model."
How Cloudflare Pay Per Crawl Works for a Data Buyer
When a crawler asks for a page, a publisher can answer with HTTP 402 Payment Required. The publisher chooses Allow, Charge, or Block for each type of crawler. Your bot is recognized through its declared identity, using Web Bot Auth. Traffic falls into three categories: Search, Agent, and Training.
Your category decides your outcome. Your identity decides whether you get a category at all. And the price, if there is one, is set per publisher.
Who Cloudflare Pay Per Crawl Actually Affects
*The table is a reading of how Cloudflare's categories are likely to apply, not a guarantee.
The fine print on the 15 September 2026 rule: Cloudflare's changelog says training crawlers and AI agents are blocked on ad pages of new domains, while search stays allowed. Cloudflare's blog adds that responsible mixed-use crawlers such as Google, Apple, and Microsoft stay allowed for search. The controls are available on every plan, existing settings carry over automatically, and settings can change.
A bot that does not identify itself may get no category at all. How that traffic is treated depends on each site's settings, and there is no public data on how many non-AI teams have been blocked. The rule is narrow on paper. In practice, the only way to know is to look at your own logs.
This week
- Search your logs for 402 and 403 responses since 15 September.
- List which of your bots send a declared identity through Web Bot Auth.
- Mark which target domains carry ads.
- Compare your block rate before and after 15 September.
What It Costs: Retries, Blocks and Paid Access
Any price you read in a blog post is a guess. Look at the bill you already have:
- Retries that multiply every request.
- Failed requests you still pay compute for.
- Heavier infrastructure to compensate.
- Engineering hours spent on a pipeline that worked last month.
Web scraping cost has always been more than the cost of a request. A blocked job still consumes compute, time, and attention.
The useful comparison is: what does blocked access cost, and what does paid access cost? A CFO will ask exactly that.
Five questions get you most of the way:
- How much is the dataset worth to the business?
- How much of your traffic is blocked or degraded today?
- How much of that is retries?
- How many engineering hours keep the pipeline alive?
- What would a license or paid access cost per month, once a publisher names a price?
It needs your own logs and one honest spreadsheet.
Three Legitimate Paths: License, Declare, or Use Public Data
Most data buyers have legitimate purposes. These paths serve them, whatever the price turns out to be.
License or pay
Respond to the 402. Sign a direct agreement. Use Pay Per Crawl, Pay Per Use, or the Monetization Gateway, which Cloudflare says entered beta on 30 September and lets sellers charge agents per request using the open x402 protocol. This is AI crawler licensing in its simplest form.
Crawl as a declared, verified bot
Give your crawler a stable identity. Use Web Bot Auth. Respect robots.txt. A publisher can only allow what it can identify.
Use public data and official sources
APIs. Public datasets. Terms of service you have actually read.
What Stays True No Matter the Price
Prices will move, and categories will be renamed. Some things won't:
- Declared, verifiable access beats anonymous access over the long run.
- The legal framework and the terms of each site still apply.
- Policies change, so the category you are placed in today may not be the one you are placed in next quarter.
That's why a quarterly audit is cheap insurance. Recheck your block rate after each Cloudflare update.
One hypothesis, labeled as one: demand for anonymous traffic infrastructure might shift toward licensing or verifiable traffic. (There is no public data to confirm it.)
FAQ
Is Pay Per Crawl the same as Pay Per Use?
No. They are complementary. Pay Per Crawl charges for access, Pay Per Use pays for what happens after, and publishers choose the model.
Does it affect Googlebot?
Search crawlers stay allowed. According to Cloudflare's blog, the "Disallow AI Training" setting leaves responsible mixed-use crawlers, such as Google, Apple, and Microsoft, allowed for search.
Does the 15 September 2026 default apply to my existing domain?
Cloudflare describes the rule as covering new domains with ads, and says existing settings carry over. Check your own dashboard.
Do I have to pay to crawl a Cloudflare site?
Not by default. It depends on each publisher's decision: Allow, Charge, or Block. There is no public data on how many publishers charge.
Is this legal advice?
No. It is a market reading based on Cloudflare's public announcements. Ask a lawyer who knows your jurisdiction.
Where This Leaves You
The default is now ask-first, and the open questions are real: prices, adoption, and how many non-AI teams were caught as collateral. What you control is how clearly your bot says who it is and how well you know your own block rate.
Which would you rather find out first: what a license costs, or what your blocks have already cost you?
Sources
- Cloudflare blog: Introducing pay per crawl, 1 July 2025
- Cloudflare changelog: New options to manage AI traffic, 1 July 2026
- Cloudflare blog: Content Independence Day, one year on, 1 July 2026
- Cloudflare blog: Have it both ways: stay discoverable in search while disallowing AI training
- Cloudflare blog: Pay Per Use, 30 September 2026