Google, Microsoft, OpenAI and Anthropic back one robots.txt AI training opt-out, but none honors it yet

The shared AI training opt-out is a one-line robots.txt signal that Google, Microsoft, OpenAI and Anthropic all said on October 30, 2026 they will honor, eventually. It is the first time the four companies have backed one mechanism for telling AI developers not to train on a website. It is also a proposal, not a standard, and for most sites it does nothing today.

What was announced

Google's announcement, posted October 30, 2026 by John Mueller, says the company plans "to support a shared AI training control in robots.txt" so that websites can "express their AI training preferences across multiple AI providers in a single step". The control is based on work from the AIPREF working group at the Internet Engineering Task Force.

The mechanism is a shared "policy token" placed in robots.txt. According to PPC Land's reading of the draft, the token is __usage_ai-training. It goes on the User-agent line of a robots.txt group in place of a crawler name, and it states a preference about how content is used, not whether a crawler may fetch it.

The underlying document is an Internet-Draft, draft-kelley-aipref-training-preference-signal-00, with Kevin Kelley of Anthropic as sole author. It is an individual submission with no formal IETF standing yet, it is aimed at the standards track, and it expires May 3, 2027 unless renewed.

What each company committed to

The four commitments differ in specificity:

• Google: if a site opts out with the shared token and does not mention Google-Extended, Google "plans to interpret that as if the publisher had disallowed Google-Extended". It says it aims "to support this new control along with other AI providers in 2027".
• Microsoft: will honor AI Training preferences in robots.txt starting February 1, 2027, and adds a separate page-level AI Grounding preference. Microsoft's post says publishers "should not have to choose between being discoverable in Bing and deciding how generative models use their work".
• OpenAI: aims to implement the signal "in the coming months" and will publish details later, according to PPC Land's summary. Existing robots.txt controls stay in place.
• Anthropic: says implementation has started, with no date, in a message to the IETF ai-control mailing list from Kelley.

Microsoft's schedule is the only one with real dates. Its new preferences take effect February 1, 2027, and it says it will stop reading the older NoArchive and NoCache tags as AI controls after March 1, 2027.

Training is not grounding

The shared token covers training only, and the draft defines training narrowly: using an asset to modify a generative model's learned parameters. Models used for classification, ranking or scoring are excluded.

Grounding, where an AI answer pulls in live web pages, is a different use. Microsoft's new AI Grounding preference works at the page level through a meta tag or HTTP header, with three values: Allow, Disallow and DisallowExtended. Microsoft warns that restrictive choices may reduce visibility in AI experiences and that "neither choice guarantees inclusion, traffic, or payment".

Google's post draws the same line from its side: sites that specifically allow Google-Extended are eligible to appear in responses and receive clicks via grounding in Gemini Apps. So a publisher who opts out of training with one line may still want to think about whether it wants to appear in answers.

What it does not do

Read the fine print and the limits are long:

• It does not block crawling. Robots.txt rules still govern whether a bot fetches a page.
• It does not affect search rankings in Google or Bing.
• It does not remove material already collected, so existing datasets and trained models are untouched.
• It has no enforcement mechanism, and PPC Land reports that OpenAI calls its participation voluntary.
• Private licensing deals may override it.
• Meta, Amazon and Apple made no commitments, according to the same report, so a large share of AI crawling is not covered by the four announcements.
• A wildcard group (User-agent: *) does not count, and no robots.txt file published today expresses the preference.

That last point is why the headline overstates the immediate effect. Until the companies ship crawler support, adding the token is a statement of intent that nobody is obliged to read. The base standard, RFC 9309, says plainly that "These rules are not a form of access authorization," so even honored rules are a request, not a lock.

Why the timing matters

Google says the proposal "is consistent with the EU Code of Practice for General Purpose AI (GPAI) Model Providers under the EU AI Act that recognizes robots.txt as a primary method for web publishers to express machine-readable preferences". That sentence is the regulatory reason four rivals agreed on something. The EU's rules for general-purpose model providers already point to robots.txt, and a shared token lets each company say it follows an industry signal rather than inventing its own.

It also lowers the cost for publishers. Today a site that wants out of AI training has to track a growing list of company-specific tokens, with Google-Extended dating to 2023. One line that several companies honor is easier to maintain and easier to audit.

What we would do

If you run a site, do not rush, but do not ignore it. Keep your existing company-specific rules. Decide your position on training and on grounding separately, because Microsoft now lets you choose independently and the trade-off is visibility versus control. Add the token when your own crawler logs show a participating company's bot has started reading it.

If you follow the policy fight, watch the IETF working group. The vocabulary beyond training is still under negotiation, and a one-author draft becomes a standard only if the companies' promises turn into shipped code. We are keeping the per-company dates in a separate tracker and will update it as each one lands.