On September 15, 2026, the cloudflare ai crawler default changes for every new domain onboarding to the network. Crawlers Cloudflare classifies as Training and Agent get blocked on pages that display ads. Crawlers classified as Search stay allowed. Existing customers keep whatever settings they already have unless they change them before that date.
That sounds like a routine settings change and it is not. It is the first time a company sitting in front of a meaningful share of the web has made a default judgment about why a bot is visiting and enforced different answers for different reasons. Robots.txt never had that vocabulary, the IETF is still trying to standardize one, and Cloudflare has shipped a working version ahead of both. This piece covers what actually changes on the date, the three categories and where they get blurry, who is affected and who is not, what the ad-supported condition really means, what to check on your own domain, and why the default matters more than the toggle. For the groundwork, see our guide to auditing your site for AI crawler access.
What the cloudflare ai crawler default actually does
Cloudflare’s announcement states the change in one sentence: "Training and Agent will be blocked by default on the pages that display ads, while Search will remain allowed by default."
Three qualifiers in that sentence do the work, and each one narrows the scope more than the headlines suggest.
| Qualifier | What it means |
|---|---|
| Training and Agent | Two of three purpose categories. Search is explicitly excluded from the block. |
| Pages that display ads | Not the whole site. The block is scoped to ad-supported pages, where automated access competes directly with the monetization model. |
| New domains | Cloudflare’s wording is “all new domains onboarding to Cloudflare.” Existing zones keep their current configuration. |
Cloudflare also says the opt-out is straightforward: a site owner who does not want the new configuration "can easily mark this in their Security settings any time leading up to September 15."
The three categories, and where the line blurs
The classification is the substantive part, and it is worth learning because other infrastructure vendors will copy it.
Search is behavior that "collects or indexes your content, so it can answer questions about it later." That is the traditional crawl-to-index bargain: you give up the content, you get referral traffic back.
Agent is "automated behavior that is acting, usually in real time, on a person’s behalf." A person asked a question, something went and fetched your page to answer it, and the fetch happened now rather than during a nightly crawl.
Training is "a crawler taking your content to train or fine-tune a model." No traffic comes back, now or later.
Cloudflare tracks further classifications beyond these three, including ads verification, feed fetching, and security testing. The company says it "strongly encourages" operators running multiple purposes to separate their crawlers so the classification can be accurate.
That last point is where the pressure lands, and it is the actual target of the policy. An operator running one undifferentiated crawler for indexing and training has an accuracy problem that is now expensive, because a bot that cannot be shown to be Search is a bot that gets treated as something else. The incentive to split crawlers by purpose is created by the default, not by a mandate.
Cloudflare has been direct about the reasoning. In TechCrunch’s reporting on the announcement, chief executive Matthew Prince framed it as a pace problem rather than a policy one: "Now that the majority of traffic on the Internet is non-human, we must go further and act faster so that a sustainable ecosystem can emerge." The same reporting notes that more than half of AI crawler traffic consists of re-fetching pages that have not changed, which is the least defensible version of the load publishers are absorbing.
Why the ad-supported condition is the interesting part
Blocking Training and Agent everywhere would have been the louder move. Scoping it to pages that display ads is the more defensible one, and it is worth understanding why.
On an ad-supported page, the business model is explicit and mechanical: a human loads the page, an impression is served, the publisher is paid. An agent that reads the page and reports back to a person has consumed the content and skipped the transaction. The harm is not theoretical and it does not require anyone to take a position on whether training is fair use.
On a page with no ads, that argument does not hold in the same way, and Cloudflare has not made it. The default is narrow because the justification is narrow. Publishers who assumed this was a blanket block should read the scope carefully before planning around it. We covered the underlying economics in what publisher traffic collapse looks like at small scale.
Who this actually affects
Four groups, in descending order of how much they need to care.
New Cloudflare domains that run ads. The default applies to you on September 15. This is the group the change was written for.
Existing Cloudflare customers. Nothing changes automatically. Your current settings persist. That is a mixed blessing: a site that has never touched its bot settings keeps a permissive configuration that predates the categories existing, which is worth revisiting on its own merits rather than because of the date.
Sites not behind Cloudflare. No direct effect, and one indirect one. If the classification scheme spreads, the practical definition of a well-behaved crawler will have been set by a private company rather than by a standards body.
AI companies running crawlers. The operational deadline is real. A crawler whose purpose cannot be determined loses access to a growing share of ad-supported pages by default rather than by decision.
What to check on your own domain
Five things, and none of them take long.
Confirm which side of the line you are on. If your zone predates September 15, you are on your existing settings, not the new default. Do not assume you inherited the protection.
Look at your current bot configuration rather than trusting memory. Most site owners set this once, if ever, and the categories are newer than the settings.
Decide the Search question deliberately. Allowing Search is the default because it is usually correct, but "usually" is doing real work there if AI answers already satisfy most of your queries without a click. Our piece on AI citations and rankings decoupling covers how to tell.
Check whether your pages are classified as ad-supported the way you expect, particularly if ads are served through a tag manager or appear on only part of the site.
Add the fourth Content Signals parameter if it fits your position. Cloudflare has added use=reference, which signals a preference for indexing and excerpting with attribution, to its managed robots.txt.
The default is the policy
The toggle has existed for a while. What changed is which way it points when nobody touches it, and that is the whole story.
Defaults decide outcomes at scale because most site owners never open the settings page. A permissive default meant the open web’s answer to "may I train on this" was yes unless you knew to say no. A restrictive default on ad-supported pages flips the burden onto the crawler operator for a specific, defensible subset of the web. Neither is neutral. Both are policy, and the second one is at least stated out loud.
The part worth watching is that this arrived from an infrastructure company rather than from a standard. The IETF’s AI preferences working group is the neutral alternative, chartered to "standardize building blocks that allow for the expression of preferences about how content is collected and processed" for AI. Both of its deliverables carry an August 31, 2026 milestone for submission to the IESG, two weeks before Cloudflare’s defaults take effect. Until that work produces something implementable, the operative definition of Search versus Training versus Agent is the one written by whoever sits in front of the most traffic. Governance by default configuration is a pattern we also traced in the CMS agent governance standard.
What this does not settle
Three things stay open, and it is worth being clear about them rather than reading the announcement as a resolution.
Classification accuracy is unproven at scale. A category system only works if the categories can be assigned correctly, and self-declaration by crawler operators is the weakest link in any such scheme.
Compensation is a separate question. Cloudflare’s pay per crawl mechanism answers a crawler either with a normal response or with an HTTP 402 Payment Required carrying a price, and it remains in private beta. Blocking by default is not the same as getting paid. The block establishes leverage; whether leverage becomes revenue for anyone other than the largest publishers is unresolved.
And enforcement across the rest of the web is not addressed by any of this. A default on one network changes conditions for the sites on that network. It does not change what happens to everyone else, which is the case for a standard rather than against Cloudflare acting.
Frequently Asked Questions
What is the cloudflare ai crawler default changing to on September 15?
For new domains onboarding to Cloudflare, crawlers classified as Training and Agent are blocked by default on pages that display ads. Crawlers classified as Search remain allowed by default.
Does this apply to my existing Cloudflare site?
No. Existing customers keep their current settings. The new default applies to domains onboarding on or after September 15, 2026. If you want the new posture, you have to select it yourself.
Can I opt out of the new default?
Yes. Cloudflare says site owners who do not want the new configuration can mark that in their Security settings at any time leading up to September 15.
What are the three crawler categories?
Search collects or indexes content to answer questions about it later. Agent is automated behavior acting in real time on a person’s behalf. Training is a crawler taking content to train or fine-tune a model. Cloudflare also tracks other classifications including ads verification, feed fetching, and security testing.
Why only pages that display ads?
Because that is where automated access most directly displaces the revenue model. A human view serves an impression and pays the publisher; an agent read does not. Cloudflare has not extended the default block to pages without ads.
What happens to a crawler that does more than one thing?
Cloudflare strongly encourages operators to run separate crawlers per purpose so classification is accurate. A crawler that cannot be identified as Search does not get Search’s default treatment.
Does this replace robots.txt?
No, it sits alongside it. Cloudflare has added a fourth Content Signals parameter, `use=reference`, indicating a preference for indexing and excerpting with attribution, to its managed robots.txt.
Is there a neutral standard for any of this?
Not yet. The IETF has an AI preferences effort aimed at exactly this problem. Until it produces something implementable, purpose-based crawler policy is defined by infrastructure vendors rather than by an open standard.
