Google plans to support a shared AI training control in robots.txt. It is one line that would let a site opt out of AI training at several AI providers at once. Google announced the plan on October 30, 2026. It aims to support it, with other AI providers, in 2027.
Nothing changes on your sites today. Google-Extended is still the control Google reads for training, and Googlebot rules still decide what appears in Google Search. But the new signal changes how a training opt-out maps onto Google’s systems. That mapping has a side effect agencies should know about before they add the line for clients.
The short version: Google says it will treat the shared opt-out as a Google-Extended disallow, unless robots.txt names Google-Extended directly. Google-Extended covers grounding in Gemini Apps as well as training. So a blanket opt-out may also take a site out of Gemini Apps answers. Keep your current rules, decide per client what they want, and put a robots.txt review on the 2027 calendar.
This post covers what Google announced, how the AI training control works and what it does not touch. It also covers what OpenAI, Anthropic and Apple document today, plus a checklist for client sites. It follows our AI crawler access audit from June with real news.
What Google announced on October 30
According to Google’s Search Central blog post, written by John Mueller, Google plans to support "a shared AI training control in robots.txt." The work is based on the AIPREF working group at the Internet Engineering Task Force (IETF).
Today, a site that wants out of training needs a separate rule for each AI company. The shared control replaces that with one "policy token" that participating providers would all read.
Google describes three parts:
- One token, many providers. A site uses the policy token to set a broad training preference across participating AI providers.
- A mapping to Google-Extended. If a site opts out with the token and does not name Google-Extended, Google plans to read it as a Google-Extended disallow.
- Company rules still work. Existing controls such as Google-Extended keep working alongside it. A site can set a broad rule and then make exceptions.
The post calls this a "draft proposal" for now. Google says it is consistent with the EU Code of Practice for general-purpose AI model providers under the EU AI Act. On timing, the post says "We aim to support this new control along with other AI providers in 2027" and promises progress updates.
The token name Google’s docs already show
Google’s blog post does not give the syntax. Its crawler documentation does. On October 30, Google’s crawling documentation changelog logged a note about "future AI training control support" on the common crawlers page.
That note, on Google’s list of common crawlers, says Google intends to support the AIPREF signal "in early 2027" as a robots.txt token. It names the robots.txt line as user-agent: __usage_ai-training.
Two cautions. First, the note shows only that user-agent line, not a full example group with allow and disallow rules. Second, the IETF work is still in draft. The AIPREF working group page lists two drafts. One defines the vocabulary. The other covers attaching preferences to content, with robots.txt as one method. Details could change before anyone ships support.
As of November 2026, the AI training control has a name. It has no confirmed behavior in production anywhere.
How the AI training control maps to Google-Extended
To see what the shared opt-out will do at Google, you need to know what Google-Extended does now.
Google’s crawler docs call Google-Extended "a standalone product token." It has no user agent of its own. Google crawls with its normal crawlers and reads the Google-Extended rule to decide how that content may be used. It controls two things.
- Training of future Gemini models that power Gemini Apps and the Vertex AI API for Gemini.
- Grounding in Gemini Apps and in Grounding with Google Search on Vertex AI. Grounding means giving the model content from the Search index at prompt time.
That second item matters. Google’s blog post says sites that "specifically allow Google-Extended are eligible to appear in responses" via grounding in Gemini Apps.
Put the pieces together. Say a client adds the shared opt-out and leaves Google-Extended out. Google says it will act as if Google-Extended were disallowed. Based on Google’s own description of Google-Extended, that would cover grounding in Gemini Apps too, not only training. A client who wants out of training but still wants to appear in Gemini Apps answers would need an explicit Google-Extended rule.
This is our reading of two Google pages side by side. Google has not published an example of that exact combination yet. Recheck it when Google ships support.
What the AI training control does not touch
Google is clear that Google-Extended does not affect Search. Its crawler docs say Google-Extended "does not impact a site’s inclusion in Google Search" and is not a ranking signal.
AI Overviews and AI Mode are part of Search. Google’s AI features documentation says Googlebot rules in robots.txt control how a site is crawled for Search. To limit what shows from a page in Search, Google points to nosnippet, data-nosnippet, max-snippet or noindex.
| Control | What it affects | Affects Google Search? | Status, November 2026 |
|---|---|---|---|
| Googlebot rules in robots.txt | Crawling for Google Search, including AI Overviews and AI Mode | Yes | Live |
| nosnippet, data-nosnippet, max-snippet, noindex | What Search shows from a page, or whether it is indexed | Yes | Live |
| Google-Extended | Gemini training, plus grounding in Gemini Apps and Vertex AI | No | Live |
Shared AI training control (__usage_ai-training) |
Training across participating providers; read as Google-Extended at Google | No, per Google’s mapping | Planned, early 2027 per Google’s docs |
The practical point: blocking Googlebot to stop training is the wrong tool. It removes the site from Search. Our post on the generative AI performance report covers the separate Search Console opt-out for AI features in Search.
What OpenAI, Anthropic and Apple document today
As of November 2, 2026, no other major provider mentions AIPREF or a shared token on its crawler page. Each uses its own robots.txt names. OpenAI’s crawler overview says a GPTBot disallow means content "should not be used in training" for its models. OAI-SearchBot is separate and governs ChatGPT search. Anthropic’s help article lists ClaudeBot for training, Claude-User for user requests and Claude-SearchBot for search, and supports Crawl-delay. Apple’s Applebot page says Applebot-Extended "does not crawl webpages" and only governs training use. We mapped these in our AI crawler access audit and our post on Cloudflare’s BotBase registry.
Who should care
Most of our readers run sites for nonprofits, associations, colleges and public agencies. For them, the AI training control is a governance question more than a technical one.
Some clients want maximum reach. A public health nonprofit may want its guidance in every AI answer. A membership association may want to keep paid research out of training sets. A university may have different rules for its news site and its library.
Today, each of those choices means a list of company-specific rules that goes stale as new crawlers appear. The shared token, if providers adopt it, makes the broad choice a single line. The hard part stays the same. Someone at the client has to decide, and you have to write it down.
A robots.txt checklist for client sites
Use this now, and again when Google ships support.
- Inventory current AI rules. Pull robots.txt for every client host and subdomain. Note each AI user agent and its rule.
- Check who writes robots.txt. CDNs and hosts may add rules. See our post on the [Cloudflare AI crawler default](/search-engine-optimization-seo/cloudflare-ai-crawler-default-block/).
- Separate the three decisions. Ask each client about training, AI search and grounding, and classic Search. They are different controls.
- Keep Googlebot open unless a client truly wants out of Google Search.
- Make Google-Extended explicit. If a client wants to appear in Gemini Apps answers, an explicit Google-Extended allow will matter later.
- Record the decision in the client’s site documentation, with the date and who approved it.
- Watch the source. Subscribe to Google’s crawling changelog RSS feed, linked on the changelog page.
- Book a 2027 review. Add the `__usage_ai-training` group only after providers confirm support and publish a full example.
Frequently Asked Questions
What is Google’s shared AI training control?
It is a planned robots.txt signal, based on IETF AIPREF work, that lets a site state one AI training preference that several AI providers can read. Google announced its plan to support it on October 30, 2026.
When will Google support the AI training control?
Google’s blog post says it aims to support it with other AI providers in 2027. Its crawler documentation says “early 2027.” No exact date is published as of November 2026.
What is the robots.txt syntax?
Google’s common crawlers page names the line `user-agent: __usage_ai-training`. It does not show a full example group yet, and the IETF drafts are not final.
Should I add the new token to robots.txt now?
There is no benefit yet, since Google and the others have not confirmed support in production. Keep your current company-specific rules and revisit in 2027.
Will opting out of AI training remove my site from Google Search?
No. Google says Google-Extended does not affect inclusion or ranking in Search, and Google plans to read the shared opt-out as a Google-Extended disallow.
Does it affect AI Overviews or AI Mode?
No. Those are part of Search. Google says Googlebot rules and preview controls such as `nosnippet` and `noindex` are the controls for AI features in Search.
Could the shared opt-out remove my site from Gemini Apps answers?
Possibly. Google-Extended covers grounding in Gemini Apps, and Google says only sites that allow Google-Extended are eligible there. Name Google-Extended explicitly if a client wants to stay eligible.
Do OpenAI and Anthropic support the shared token?
Their crawler pages do not mention it as of November 2, 2026. OpenAI uses GPTBot for training, and Anthropic uses ClaudeBot. Block those by name if a client wants out now.
Does blocking Google-Extended stop Google from crawling my site?
No. Google-Extended has no user agent of its own. Google crawls with its usual crawlers and uses the Google-Extended rule to decide how the content may be used.
