Website owners are set to get one line in robots.txt that tells several AI companies not to train on their content. On October 30, 2026, Google and OpenAI said they will honor a shared AI training preference based on work at the Internet Engineering Task Force (IETF). The Verge reported that "OpenAI, Google, Microsoft, and Anthropic say they will adopt a new standard." It is an opt-out from AI training, not a block on crawlers.

#What did Google announce?

Google's post says: "Today, we're announcing our plans to support a shared AI training control in robots.txt." It adds that this "will help websites express their AI training preferences across multiple AI providers in a single step and is based on work from the AIPREF working group at the Internet Engineering Task Force (IETF)."

Instead of separate rules for every company, "website owners will be able to use a shared 'policy token' within robots.txt to set broad preferences across participating AI providers." Google explains how it will read the token: "When a website uses this policy token to opt out of training in its robots.txt file (and Google-Extended is not explicitly specified), Google plans to interpret that as if the publisher had disallowed Google-Extended."

Google-Extended is the AI training control Google has offered since 2023, and it stays. "Existing company-specific controls will continue to function alongside this new joint signal," Google says. It also notes: "Websites that specifically allow Google-Extended are eligible to appear in responses and receive clicks via grounding in Gemini Apps."

On timing, Google says: "We aim to support this new control along with other AI providers in 2027, and will provide updates on our progress." It calls the plan a "draft proposal" and says it is "consistent with the EU Code of Practice for General Purpose AI (GPAI) Model Providers under the EU AI Act."

#What did OpenAI say?

OpenAI's post says: "Today, we’re announcing that OpenAI will honor a common signal" in robots.txt. It describes the effort as one that "makes it easier for publishers and website operators to communicate the same preference to multiple participating providers at once." Before this, it says, "website operators would identify individual crawlers and maintain separate rules for each and every provider."

OpenAI gives no date. It says: "We aim to implement the signal in the coming months, and will share another update with details about our implementation when we do so." Its current controls stay too: "This new signal will not replace the controls we already provide through robots.txt." OpenAI adds: "We encourage other AI providers to honor the signal."

#Where do Microsoft and Anthropic stand?

Microsoft's details here come from The Verge. It reported that "Microsoft, however, will offer more granular controls that will give publishers the ability to opt out of AI training or AI grounding, the process of pointing an AI model to a verified source of information to reduce hallucinations." The Verge also wrote: "Microsoft notes that websites will still appear in Bing’s search results regardless of their preferences." Per The Verge, "Google and Microsoft plan to support it next year," meaning 2027.

For Anthropic, The Verge said it "has already adopted the signal." Anthropic's Kevin Kelley, who wrote the IETF draft, put it more carefully in a post to the IETF mailing list: "I'm also excited to share that Anthropic has already started implementing support for this draft." He added: "It'll take us some time to add support across our services." Kelley said the token "complements our existing crawler controls rather than replacing them." Anthropic's Help Center describes its existing training opt-out: "When a site restricts ClaudeBot access, it signals that the site's future materials should be excluded from our AI model training datasets."

#How do you use the token?

The token comes from the IETF Internet-Draft, not from Google or OpenAI. The draft says: "This document defines the policy token __usage_ai-training." It goes in the User-Agent line of a robots.txt group. To opt a whole site out of AI training, the draft's rules mean you add these two lines:

User-Agent: __usage_ai-training

Disallow: /

The draft's own example also adds an exception, so one folder stays available for training:

Allow: /press-kit/

A few rules from the draft matter:

  • Disallow means no. "If the lookup finds that the path is disallowed, a preference against the use described by the policy token has been expressed for that path."
  • No token, no preference. "If no group contains the __usage_ai-training policy token (i.e., there is no User-Agent: __usage_ai-training line in the file), then no preference has been expressed."
  • Wildcards do not count. "However, wildcard groups (i.e., User-Agent: *) MUST NOT be interpreted as matching the policy token."
  • It is about training. The draft defines it as "AI Training: Using an asset to modify the learned parameters of a generative AI model."

#Does this block AI crawlers?

No, not on its own. The draft says: "This policy token is not a directive for crawling behavior." Kelley wrote: "This is a preference about use, not a traffic-management signal." The draft also says "Private agreements MAY override preferences expressed via this mechanism."

The Verge described the token differently, writing that the companies "say they will restrict their AI crawlers from accessing a site if they encounter the shared token." The companies' own pages describe a training preference, and Google and OpenAI keep their separate crawler controls in place.

The draft is an individual Internet-Draft. Its Datatracker page says: "This I-D is not endorsed by the IETF and has no formal standing in the IETF standards process."

For related coverage, see how OpenAI trains frontier models offline and Google's Connected Apps Platform preview for Gemini and Search.

Our analysis: One shared switch is a real improvement for site owners, who have had to track each company's crawler names. But it is a preference, not a lock. It covers training, not crawling, and private deals can override it. Timing is uneven too: Google points to 2027, OpenAI gives no date, and Anthropic says support across its services will take time. Site owners who want an opt-out that all four companies honor should keep their company-specific rules and add the shared token alongside them.

#What is still unclear?

  • OpenAI's date. OpenAI says only "in the coming months."
  • Microsoft's details. Microsoft's own start date and grounding syntax are not in the sources we used.
  • Anthropic's coverage. Kelley says support across Anthropic's services will take time.
  • Precedence. Only Google spells out how the shared token interacts with its own control.