Cloudflare shipped two things eight days apart in August 2026, and almost every write-up treated them as separate product news. Bot Preference Sync landed on August 21 and generates your robots.txt from your dashboard settings. BotBase for Operators landed on August 28 and gives bot operators a managed process for getting into Cloudflare’s directory of known bots.
Put them side by side and they are one thing. The first takes the declaration side of crawler control and turns robots.txt from a file you author into a file that gets generated. The second takes the identity side and turns "is this a real crawler" into an application with a review queue. Both halves of a system that ran on convention for thirty years now run through one company’s dashboard, and the default flip we covered on September 15 is what makes the settings consequential. This piece covers what happened to robots.txt, what Cloudflare now writes into it, how the BotBase submission process works, where the registry critique is right and where it is imprecise, what to do as a site owner, what to do as a bot operator, and the question none of it answers.
robots.txt did not get replaced, it got demoted
The framing to avoid is that robots.txt is dead. It is not. It is still there, still fetched, still parsed. What changed is its position in the stack.
Cloudflare’s own description is precise: "Cloudflare generates or updates your robots.txt based on your configuration, so what you say to the world and what you enforce at the edge are kept in sync." The file is now an artifact of a setting rather than the place the decision is expressed. If you want to know your own crawler policy, the answer is no longer in the file; the file is a rendering of the answer.
For anyone who has ever found a robots.txt that contradicted the actual edge configuration, this is an improvement, and it is worth saying so plainly. The two have drifted apart on more sites than anyone would like to count. Generating one from the other eliminates a whole class of quiet failure.
What it actually writes
Three categories, each mapping to directives in the generated file. Search can be set to allow, block on pages with ads, or block everywhere. Agent gets the same three options. Training gets a disallow that writes a no-training preference into robots.txt.
Two implementation details matter more than the categories. First, if you already have a robots.txt, Cloudflare’s directives are prepended rather than substituted, so existing Disallow rules survive. Second, availability is total: every tier from Free to Enterprise, enabled by default for new customers, with existing customers on the legacy managed robots.txt feature prompted to confirm during transition. You can turn it off entirely and keep a hand-written file.
"Enabled by default for new customers" is the phrase to sit with. A large number of sites will ship a Cloudflare-generated robots.txt without anyone deciding to, which is how defaults work and why they are worth auditing.
BotBase is an admissions process
BotBase for Operators is the side that gets less attention and deserves more, because it is where the harder question lives: who gets to count as a legitimate bot?
An operator now declares three things. Behavior, meaning what the bot does, chosen from categories like indexing for search, acting as an agent, collecting data, training models, or supporting SEO tools. Content use, expressed in the Content Signals model, in the form of declarations like search=yes, ai-train=no, use=reference. And operator type, meaning whether they run the bot directly or act as an intermediary platform.
Submissions then sit in one of three states: waiting for review, accepted, or rejected with a stated reason. Cloudflare says the process previously ran where "every submission followed the same fully manual path," and that it now runs automated checks first, verifying an IP list, confirming reverse DNS, or validating a Web Bot Auth signature, escalating to a human only when needed. The motivation is volume: submissions are up roughly 7 times since 2023.
The result is published as a searchable directory through Cloudflare Radar, which site owners use to decide what to allow.
The registry critique needs to be more precise than it usually is
The easy version of this argument is that Cloudflare is replacing an open standard with a private registry. That version is wrong in an important way, and getting it right sharpens the actual concern.
The building blocks here are open. Content Signals is a published vocabulary. Web Bot Auth is a cryptographic identity mechanism developed in the open. Reverse DNS verification is ancient and universal. None of that is proprietary, and a competitor could implement the same verification stack tomorrow.
What is not open is the decision. Acceptance, rejection and the reasons for rejection are Cloudflare’s, applied by Cloudflare’s criteria, to a directory Cloudflare publishes. There is no published appeals path, no external review, and no obligation to justify a criterion. That is the thing worth watching, and it is a governance question rather than a standards question. The protocols are shared. The gate is not.
It is also fair to note what the old system offered instead, which was nothing. robots.txt never verified anybody. A crawler could claim any user-agent string it liked, and plenty did. Replacing an unenforceable honor system with a verified directory is a real gain in exchange for a real concentration. Both halves of that trade are true, and coverage that reports only one of them is not useful.
What this means if you run a site
Check what your robots.txt currently says, and whether Cloudflare is generating it. This is a thirty-second job and a surprising number of people will find an answer they did not choose. Our AI crawler access audit still describes the right method; the change is that the file may now be an output rather than an input, so fixing it by editing the file is the wrong move if a dashboard setting is upstream of it.
Then decide the three categories deliberately rather than accepting them. Search, Agent and Training are genuinely different questions with different business answers, and the most common mistake is treating them as one dial. Blocking training while allowing search is coherent. So is the reverse, for some publishers. Blocking all three because it felt safest is a decision that should be made on purpose.
If you want to be cited by AI systems rather than merely crawled, the Agent setting is the one that matters most, and it interacts with everything in writing for AI citation. Blocking agent traffic and then wondering why you are absent from AI answers is an avoidable contradiction.
What this means if you run a bot
Read the behavior taxonomy before you submit, because the categories are the vocabulary you are now required to speak. If your crawler does two things, you will have to say which one it is, and that classification travels with you into every site owner’s allow decision.
Declare content use honestly under Content Signals. A declaration of ai-train=no that turns out to be false is now a checkable claim attached to a verified identity, which is a different risk profile than an unverifiable user-agent string ever carried.
And implement Web Bot Auth if you can, because signature validation is the path that clears automatically. The alternative is an IP list you have to keep current, or a human review queue.
The part still unresolved
Our September 15 default post ended on compensation, and nothing this month moved it. A verified identity plus a declared purpose plus an enforced preference still adds up to permission, not payment. Pay per crawl remains in private beta.
Cloudflare added a third piece on August 31, after this post’s two. Adaptive Intelligence is a detection engine inside Bot Management that retrains on live traffic and issues what Cloudflare calls disposable rules, deployed and retired at random intervals so they never sit still long enough to be reverse-engineered. It is aimed at credential stuffing and scraping rather than at declared crawlers, and it is an Enterprise feature reached by enabling Auto Update Machine Learning. It belongs to the enforcement half of this story rather than the declaration half, but it lands in the same three-week window, and the pattern is consistent: identity, preference and enforcement all moving behind one dashboard.
What did move is legibility. Before this month, a site owner facing unfamiliar crawler traffic had a user-agent string and a guess. Now there is a directory entry with a declared purpose, a verified identity and a stated content-use policy. That is a genuinely better position to negotiate from, and the negotiation still has not started. Our technical SEO checklist treats robots.txt as a file you write, which is one more assumption from earlier this year that needs revisiting.
Frequently Asked Questions
What is BotBase?
BotBase is Cloudflare’s directory of known bots and agents, published through Cloudflare Radar and used by site owners to decide what traffic to allow. BotBase for Operators, released August 28, 2026, is the dashboard where bot operators submit entries, track review status, and edit them, along with a structured model for declaring what their bot does and how it uses content.
Does Bot Preference Sync overwrite my robots.txt?
No. Cloudflare prepends its generated directives to your existing file, so Disallow rules you already have are preserved. You can also disable the feature entirely and keep a fully custom robots.txt. It is available on every tier from Free to Enterprise and is enabled by default for new customers.
What are the Search, Agent and Training categories?
They are the three crawler purposes Cloudflare lets you set policy for separately. Search and Agent each offer allow, block on pages with ads, or block everywhere. Training offers a disallow that writes a no-training preference into robots.txt. They are distinct business decisions and are worth setting individually rather than as one dial.
What does a bot operator have to declare?
Three things. Behavior, meaning what the bot does, from categories including indexing for search, acting as an agent, collecting data, training models and supporting SEO tools. Content use, expressed in the Content Signals model, for example search=yes, ai-train=no, use=reference. And operator type, meaning whether they run the bot directly or operate as an intermediary platform.
How does Cloudflare verify a bot is what it claims?
Automated checks run first, verifying an IP list, confirming reverse DNS, or validating a Web Bot Auth signature, with escalation to human review when needed. Cloudflare says this replaced a process where every submission followed the same fully manual path, prompted by submission volume rising roughly 7 times since 2023.
Is Cloudflare replacing an open standard with a proprietary one?
Not exactly, and the distinction matters. The components are open: Content Signals is a published vocabulary, Web Bot Auth is an open identity mechanism, and reverse DNS verification is universal. What is not open is the admission decision. Acceptance, rejection and the criteria behind them belong to Cloudflare, with no published appeals path or external review.
Does any of this get publishers paid?
No. Identity, declared purpose and enforced preference together produce permission, not compensation. Cloudflare’s pay per crawl mechanism, which answers a crawler with either a normal response or an HTTP 402 carrying a price, remains in private beta. What improved this month is the ability to see who is asking and why.
What should I check on my own site today?
Fetch your robots.txt and compare it against what you believe your policy is. Determine whether Cloudflare is generating it, because if so, editing the file directly is the wrong repair and the dashboard setting is upstream. Then set Search, Agent and Training deliberately, particularly Agent if you care about appearing in AI-generated answers.
